H.264 : VTDecompressionSessionDecodeFrame fails with code 12909

Question

I'm displaying RTSP streams from IP cameras. I want to replace this method avcodec_decode_video2 with VideoToolbox framework. I'm kinda new with video data processing so I'm a little lost...

I've tested both solutions from here and here, but I end up with a failure for this method VTDecompressionSessionDecodeFrame.

Error Domain=NSOSStatusErrorDomain Code=-12909 "The operation couldn’t be completed. (OSStatus error -12909.)" - -12909

I suspect this is coming from the session because if I call this VTDecompressionSessionCanAcceptFormatDescription right after creating the session, the result is NO.

Here is how is created my session :

VTDecompressionOutputCallbackRecord callback;
callback.decompressionOutputCallback = decompressionSessionDecodeFrameCallback;
callback.decompressionOutputRefCon = (__bridge void *)self;
NSDictionary *destinationImageBufferAttributes =[NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithBool:YES],(id)kCVPixelBufferOpenGLESCompatibilityKey,[NSNumber numberWithInt:kCVPixelFormatType_32RGBA],(id)kCVPixelBufferPixelFormatTypeKey,nil];
OSStatus status = VTDecompressionSessionCreate(kCFAllocatorDefault, _formatDesc, NULL,
                                                   (__bridge CFDictionaryRef)destinationImageBufferAttributes,
                                                   &callback, &_decompressionSession);

I'm also not sure about the data to use. I have a choice between AVPacket.data, AVCodecContext.extradata, AVPacket.buf->data. Can anyone tell me which one should I use (maybe there is another one) ?

Thx in advance,

PS.: This is my first post ; let me know if it is not correctly presented/explained.

Answer 1

I've succeeded to get one successful status for this method. Turns out that I had indexes issues and the code I pasted had some bugs (at least, it wasn't adapted for my situation, specially for managing NALU types 1 and 5).

I have another issues because all the other calls returns a failure but I think I must post another question.