pyav / libav / ffmpeg what happens when frame from live source are not processed fast enough

Question

I am using pyav to process a live RTSP stream:

import av
import time

URL = "RTSP_url"
container = av.open(
            url, 'r',
            options={
                'rtsp_transport': 'tcp',
                'stimeout': '5000000',
                'max_delay': '5000000',
            }
        )

for packet in self.container.demux(video=0):
    for frame in packet.decode():
        # do something
        time.sleep(10)

What happens if I do something too slow? Are frames / packets dropped or are they buffered?

I guess the same question would apply to libav or ffmpeg.

Answer 1

In my experience gstreamer could store in a buffer old frames and return them even minutes later. Not sure if PyAv would do the same.

Answer 2

tcp is a guaranteed delivery protocol with built-in flow control. If you do not process the incoming data as fast as it is received, the tcp stack will buffer the data until its buffers are full at which time the tcp protocol will let the sender know that it cannot receive any more data. If this continues, the sender's output buffers will eventually fill up and then it is up to the sender to decide what to do.

An IP camera at that point may throw frames away or it may even drop the connection. Most IP cameras also use a keep-alive mechanism typically via RTCP packets sent over the RTSP stream. The camera may send Sender Reports and the receiver should send back Receiver Reports. If the camera does not get a Receiver Report within a timeout, it will drop the connection. I would have to assume that either the av library or ffmpeg is doing that.

You probably do not want to do time.sleep(10).

If you really feel that you need to discard packets, then you could examine your packets before calling decode to see if you are falling behind. If you are getting too far behind, you can discard packets that are not key frames until you catch up. The effect will be that the video will have jumps in it.