WebCodec: how to flush without restarting from key frame

Question

In WebCodec, if I do a decoder.flush() to ensure that all frames are decoded, I need to restart after with a key frame (i.e. not a delta frame). But this is really annoying as my goal is to decode a video in real time to play it at x1 speed, therefore since I decode frame by frame I do not always restart on a key frame:

decoder.configure(…);
decoder.decode(frame[0]); // this is a key frame: ok
decoder.flush(); // to obtain the decoded frame
decoder.decode(frame[1]); // this is NOT a key frame ==> ERROR!

I also tried not to use flush at all, but then it seems like it waits to see multiple frames before starting the decoding… which I don't want of course.

Another possibility I can imagine is to decode in advance the 250 next frames in one go (until finding a new key frame), but then it seems like quite dirty… Is there a better option I am missing?

Answer 1

So with the great advices of sanderstan here, I realized that all codecs might differ, and that the driver used to decode might also differ, even on the same browser (e.g. for license issues, I think that firefox uses the decoder provided by the system to decode multiple codecs). For instance, some codecs might need 10 frames to output a first frame; this can notably be explained because some frames need to know the n-1-th and the n+1-th frame to be decoded, and sometimes the order of the frame in the video does not match the order of the final frames (order of presentation)

Moreover, flush should only be used when reading the last frame of a video. So the trick instead is to send many decode, until:

• either we have enough outputted frames
• or the queue of unprocessed messages is too large (say decodeQueueSize is bigger than 3), meaning that the decoder struggles to process the frames. In that case, we can wait for this queue to be smaller (cf for instance the dequeue event) before adding new elements.

The best example I could find is https://webcodecs-samples.netlify.app/audio-video-player/audio_video_player whose code is here https://github.com/w3c/webcodecs/tree/main/samples/audio-video-player