How to extract key-frames closest to given frame numbers from H264 video with ffmpeg

Question

I know how to extract a set of frames as jpg files from a video using ffmpeg if you know the frame numbers (given below)

Extracting Frames: [40, 59, 73, 110]
/usr/bin/ffmpeg -y -hide_banner -nostats -loglevel error -i /home/pi/movie.mp4 -vf select='eq(n\,40)+eq(n\,59)+eq(n\,73)+eq(n\,110)',scale=640:-1 -vsync 0 /tmp/%04d.jpg

That will extract frames [40, 59, 73, 110] as files /tmp/0000.jpg, /tmp/0001.jpg, etc.

I also know how to extract all key frames for a given time interval:

ffmpeg -ss <start_time> -i video.mp4 -t <duration> -q:v 2 -vf select="eq(pict_type\,PICT_TYPE_I)" -vsync 0 frame%03d.jpg

That will get all I-frames from start_time through start_time+duration.

But what I would like to do is give a list of frame numbers and have ffmpeg extract the closest key-frames to each frame-number. Is there a way to do this with ffmpeg or would I have to write my own program ontop of libavcodec to do this?

Answer 1

First we need some test data. Note you can skip to the FFmpeg commands if you are doing this yourself; this is just how I came about the final commands:

ffprobe -select_streams v -show_entries frame=pict_type -of flat \
'Breaking Bad 5x4 Fifty-One.mp4' > pict_type.txt

Now let us find the largest window:

#!/usr/bin/awk -f
BEGIN {
  FS = "[.\42]"
}
$5 != "I" {
  pa++
}
$5 == "I" {
  qu = pa > qu ? pa : qu
  pa = 1
}
END {
  print qu
}

Running this yields 240, which means the radius is 120. If our target frame is 1000:

ffmpeg -i 'Breaking Bad 5x4 Fifty-One.mp4' \
-vf "select='eq(pict_type,I)*lt(abs(n-1000),120)'" -frames 1 outfile.jpg

If our targets frames are 1000 and 2000:

ffmpeg -i 'Breaking Bad 5x4 Fifty-One.mp4' \
-vf "select='eq(pict_type,I)*(lt(abs(n-1000),120)+lt(abs(n-2000),120))'" \
-frames 2 -vsync 0 %d.jpg

http://ffmpeg.org/ffmpeg-utils.html#Expression-Evaluation

Answer 2

If speed is the main concern, you can opt to skip non-keyframes at the demuxer stage.

ffmpeg -discard nokey -i video.mp4 -q:v 2 -vf select="eq(pict_type\,PICT_TYPE_I)" -vsync 0 frame%03d.jpg

This should provide a speed up of 20-100x.

If you need to extract all keyframes within a certain radius with the discard option, use

ffmpeg -discard nokey -i video.mp4 -q:v 2 -vf select="eq(pict_type\,PICT_TYPE_I)*(lt(abs(t-14),3)+lt(abs(t-107),3)+lt(abs(t-2113),3))" -vsync 0 frame%03d.jpg

Here, keyframes with a radius of 3 seconds of times t = 14s, 107s and 2113s will be selected.

You can't reference n with discard since the numbering will be wrong --> ffmpeg is only sending keyframes to the filter and n represents the count of the filtered frames. So, all variables and values are in seconds. If your video is constant frame rate, then t is simply n/ fps