I have a problem with decoding IP camera stream (h.265) using ffmpeg libraries.
I use Live555 to receive RTP payload.
mMediaSession->readSource()->getNextFrame(mVideoBuffer.data(),
mVideoBuffer.size(), Stream::Static_PayloadRead, this, Stream::Static_StreamClose, this);
First two bytes (after the start code 0001) has the nal_unit_header. I am retrieving the type, from bits 1-6 ( ( NALU[0] >> 1 ) & 0x3F ). Then I am processing the data depending on the NALu type:
enum NalUnitType
{
NAL_UNIT_CODED_SLICE_TRAIL_N = 0, // 0
NAL_UNIT_CODED_SLICE_TRAIL_R, // 1
NAL_UNIT_CODED_SLICE_TSA_N, // 2
NAL_UNIT_CODED_SLICE_TLA, // 3
NAL_UNIT_CODED_SLICE_STSA_N, // 4
NAL_UNIT_CODED_SLICE_STSA_R, // 5
NAL_UNIT_CODED_SLICE_RADL_N, // 6
NAL_UNIT_CODED_SLICE_DLP, // 7
NAL_UNIT_CODED_SLICE_RASL_N, // 8
NAL_UNIT_CODED_SLICE_TFD, // 9
NAL_UNIT_RESERVED_10,
...
NAL_UNIT_CODED_SLICE_BLA, // 16
NAL_UNIT_CODED_SLICE_BLANT, // 17
NAL_UNIT_CODED_SLICE_BLA_N_LP, // 18
NAL_UNIT_CODED_SLICE_IDR, // 19 // Current name in the spec: IDR_W_DLP
NAL_UNIT_CODED_SLICE_IDR_N_LP, // 20
NAL_UNIT_CODED_SLICE_CRA, // 21
NAL_UNIT_RESERVED_22,
...
NAL_UNIT_VPS, // 32
NAL_UNIT_SPS, // 33
NAL_UNIT_PPS, // 34
NAL_UNIT_ACCESS_UNIT_DELIMITER, // 35
NAL_UNIT_EOS, // 36
NAL_UNIT_EOB, // 37
NAL_UNIT_FILLER_DATA, // 38
NAL_UNIT_SEI, // 39 Prefix SEI
NAL_UNIT_SEI_SUFFIX, // 40 Suffix SEI
...
NAL_UNIT_INVALID,
};
If buffer doesn't have the start code, I am adding 0x00000001 at the start of the payload data, before sending it to FFmpeg. Camera is sending these NALu (in the same order):
- HEVC_NAL_VPS,
- HEVC_NAL_SPS,
- HEVC_NAL_PPS,
- HEVC_NAL_IDR_W_RADL,
- HEVC_NAL_IDR_W_RADL,
- HEVC_NAL_IDR_W_RADL,
- HEVC_NAL_TRAIL_R,
- HEVC_NAL_TRAIL_R ...
My solution works partially as I have 1/3 of image decoded. The other 2 HEVC_NAL_IDR_W_RADL slices get FFmpeg error: "First slice in a frame missing". If I lower the stream resolution I have 1/2 image with one additional HEVC_NAL_IDR_W_RADL slice.
Similiar code works with H.264 stream so I know (I hope) that Live555 and FFmpeg code should be fine.
Live555 doesn't reassemble the I frames (http://lists.live555.com/pipermail/live-devel/2016-September/020244.html)
Is there a specific way to reassemble frame send in multiple slices. I have even tried to assemble the frame as it would be Fragmented Unit NAL type 49 (How to depacketize the fragmented frames in RTP data (over UDP) for H265/HEVC?)
Many Thanks
Its ok to just stitch them together. Solution described below works for me with FFmpeg + Live555.
For I-slices I do:
And for non-I slices:
To know if a slice is the first slice of its frame you could check the
NALU[2] & 0x80, it should be non-zero only for initial slice. Live555 guarantees to give slices in their correct order. So after you encounter a slice which starts a new frame, just feed previously stitched slices to FFmpeg HEVC decoder and start stitching a new frame.PS: It also works well for other video processing API's, like Intel's MediaSDK. Despite said most of codecs should be able to consume separate NAL units instead of whole frames, I didn't manage to make it run that way for neither FFmpeg nor Intel. But described solution worked well for both of them. Still I think there might be a flag somewhere or a way to make them work with separate NALUs. It would be great if anyone could provide information on if that is possible at all and how to make it work if yes.
PPS: It works the same way for H264, but we should check
NALU[1] & 0x80instead.