So back in March I was asked if I would donate my childhood VHS tapes to a kid that had absolutely NO DISNEY MOVIES. My parents were well-off back then and I had every Disney video imaginable. I couldn't fathom being a kid with no Disney. So before I gave away my childhood, I wanted to preserve the content of the VHS tapes in digital form using my ADS Tech DVD Xpress DX2 box.
Most of the tapes were in good condition and some of the tapes had severely degraded magnetic fields. Bummer. But one thing I noticed while ripping these VHS tapes was that some of them had been recorded-over with the nightly news or some TV show...
I was pissed at first that my timeless childhood memories were replaced by the time-sensitive events of the day. But then came a more interesting aspect to this: the commercial breaks.
It was as if I went back in time and sampled the pop culture. Seeing things like: Beanie Babies, Builder's Square, 84 Lumber, Y2K, Oldsmobile, Plymouth, Datsun, and other things in the commercials really brought back memories of days gone by. The haircuts and clothes were interesting too.
When I wax nostalgic like that, there can only be one solution: preserve these ads. So how to go about that...
Step one was to get the analog information from the VHS tapes to a digital format I could work with. Simple enough. I used the DVD Xpress box and the software that came with it to rip the videos in real time to MPEG files.
Step two was a bit more complicated. How will I segment these MPEG files – separating the commercials from the show segments? There are plenty of softwares out there that will edit-out the commercials from recorded video, but they mostly just discard them. If you know anything about this blog, there is a BASH script involved and there are questions to be asked.
Question: How can you characterize a video segment or commercial?
Answer: Each video segment is typically delimited by a few solid black frames.
Question: How can I tell which frames are solid black?
Answer: I tried several methods including analyzing every single frame of a ripped VHS tape. The solid colored frames tended to have a lower bitrate and a lower frame size. This method took FOREVER! A 6 hour scratch tape took just about 8 hours to analyze frame-by-frame. Then I discovered that ffmpeg has a video filter for that and would tell me which frames are black within a certain threshold of dark pixels.
Question: OK, you said there was a script involved. Where is it?
Answer: Here oh impatient one:
The Script
#!/bin/bash
#Some housekeeping
filename=`echo $1 | cut -d. -f1`
ext=`echo $1 | cut -d. -f2`
rm -rf $filename
mkdir $filename
mkdir $filename/segments
#Log ALL THE THINGS!
ffmpeg -threads 8 -i $1 -vb 1000 -vf blackframe -f mpeg $filename/out.$ext 2> $filename/$filename.log
#Get just the black frames
grep "pblack" $filename/$filename.log | sed 's#\[blackframe.*\]\ ##g' | awk '{ print $1, $5 }'>> $filename/black_frames.log
export IFS='
'
start_pos=0
end_pos=0
for i in `cat $filename/black_frames.log`
do
if [ $i != 'frame= size=' ]
then
timestamp=`echo $i | cut -d: -f3`
if [ "$start_pos" == 0 ] && [ "$end_pos" == 0 ]
then
start_pos=$timestamp
elif [ "$start_pos" != 0 ]
then
end_pos=$timestamp
t=`echo $end_pos $start_pos | awk '{ print $1-$2 }'`
if [ `echo $t | cut -d. -f1` -gt 10 ]
then
echo $start_pos $end_pos $t
ffmpeg -ss $start_pos -t $t -i $1 -vcodec copy -acodec copy $filename/segments/$start_pos.$ext
start_pos=$end_pos
fi
else
echo "$i"
fi
fi
done
Download it HEREHow It Works
First, the script takes an argument of a file name So if I had a file named “My_Little_Pony.mpg”, I would execute the script with “./get_segments.sh My_Little_Pony.mpg”.
Then the script gets the file name and the file extension of the file being used and creates a directory structure under the current directory. This is just to keep things nice and tidy. I've seen where a 6 hour scratch tape can produce hundreds of segments. It's best to have this off in it's own folder along with the other stuff that will be explained later.
The script will then go through the file provided frame-by-frame and determine whether or not a frame is solid black. It will log all output in the directory created by the stuff in the paragraph above.
When that's done and over with after what seems like forever - really about a half hour for two hours of video – the black frames detected are parsed out of the log file into another log file called black_frames.log.
It then goes through each line of black_frames.log and uses each line as a stop or start point for extracting video segments and commercials into the “segments” folder.
When it's done with that, watch your commercials!
Issues
False Positives
With shows that have an abundance of dark frames or were actually shot at night, there will be a ton of false positives. Star Trek was a wealth of solid black frames. This leads to a lot of small segments and not a whole lot of large ones. There really is no way to correct this behavior.
Levels
If the whole video is over saturated with brightness, even frames that are solid black will appear to ffmpeg as being solid gray. This doesn't jive. You will have to adjust your video recording during the ripping process or by some other means to get the darkest black frames possible.
Final Thoughts
After I get through all of the many scratch tapes that are laying around, I'm going to start posting the interesting ads on YouTube for those that experience a similar nostalgia. I'm not too concerned with copyright in this case. It's advertisement It's something the company put out there for others to look at. Why would they come after me for giving their old products some sunlight?
I'm wondering if Jason Scott over at archiveteam has thought about this or if it's even within the mission of that organization. Maybe it's just me.
