How do I handle packet loss when recording video peer to server via WebRTC

Question

We are using the licode MCU to stream recorded video from Google Chrome to the server. There isn't a second instance of Google Chrome to handle the feedback and the server must do this.

One thing that we have encountered is when there is packet loss frames are dropped and the video gets out of sync. This causes very poor video quality.

In ExternalOutput.cpp there is a place where it detects that the current packet of data received has not incremented monotonically. Here you can see that it drops the current frame and resets the search state.

I would like to know how to modify this so that it can recover from this packet loss. Is submitting a NACK packet on the current sequence number the solution? I've also read that there is a mode where Google chrome submits RED packets (redundant) to deal with the packet loss.

Answer 1

Media processing apps has two principal different layers:

1. Transport layer (RTP/RTCP)
2. Codec layer

Transport layer is codec independent and deal with RTP/generic RTCP packets. On this layer there are couple of mechanisms for struggling with packet lost/delay/reordering:

1. Jitter Buffer (Handles packet delays and reordering)
2. Generick RTCP Feedbacks (Notifies source peer of packet lost)

On codec layer there are also couple of mechanisms for struggling with quality degradation:

1. Codec Layer RTCP Feedbacks
2. Forward error correction (FECC/RED)

To overcome Licode imperfections you should:

1. First of all it ignores any packet delays and reordering. So, you should implement mechanism (Jitter buffer), which will handle packet reodering/network jitter and determine packet lost (Probably, you could reuse webrtc/freeswitch mechanisms)

2. When your app determines packet lost, you should send feedback (RTCP NACK) to remote peer

3. Also you should try to handle ffmpeg (used for decoding video and saving it to file) decoding errors and send FIR (Fast Intra Request)/PLI to remote peer for requesting keyframes in case of errors.

Take a note, that p.2,3 requires proper explicit negotiation (via SDP).

Only after passing all this cases you could take a look to FECC/RED, because it's definetely more dificult to handle and implement.