Copying Image Buffer Output From Video Frame Decompression Callback

Question

I am new to Swift and need to utilize VideoToolBox API for H264 decoder. My application is written with Flutter SDK and am utilizing MethodChannel() class for invoking Swift functions. To decode frames, function VTDecompressionSessionDecodeFrame() is utilized. When the decompression session is done, a callback receives the decoded frame and status. First I need to copy the status to observe if I'm even providing the correct NAL/payload. The problem I'm facing is copying the results given to the callback, then to the function which called VTDecompressionSessionDecodeFrame(). I need to return the status/image to the calling function so that I may create a response for the MethodChannel()

My first hacky attempt at this issue was to allocate a class variable that would ultimately contain the data of imageBuffer variable that is provided to the callback. But this did not work because of compiler error. I tried allocating memory and copying each element of imageBuffer to the class variable.

In Flutter, there is the ValueChangeNotifier() class and was thinking could utilize a similar class in Swift, but this doesn't seem like the correct implementation.

The following code is from AppDelegate.swift file. I've provided the minimum code necessary to highlight the issue.

@UIApplicationMain
@objc class AppDelegate: FlutterAppDelegate {
  override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
      let controller : FlutterViewController = window?.rootViewController as! FlutterViewController
      let decodeChannel = FlutterMethodChannel(name: "adhoc.flutter.dev/decode",
                                       binaryMessenger: controller.binaryMessenger)
      
      decodeChannel.setMethodCallHandler( { [weak self]
          (call: FlutterMethodCall, result: @escaping FlutterResult) -> Void in
          
          guard call.method == "decodeFrame" else {
              result(FlutterMethodNotImplemented)
              return
          }
          self?.decodeFrame(call: call, result: result)
      })
      
    GeneratedPluginRegistrant.register(with: self)
    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }
    
    private func decodeFrame(call: FlutterMethodCall, result: FlutterResult) {\
        var status: OSStatus = 0
        // Receive encoded frame from MethodChannel
        // convert to UInt8 array
     
        var callback = VTDecompressionOutputCallbackRecord()
        callback.decompressionOutputCallback = { (outputRefCon, frameRefCon, status, infoFlags, imageBuffer, timeStamp, presentationDuration) in
          print("done")
            // How may I copy status/imageBuffer to caller of VTDecompressionSessionDecodeFrame?
            }
            
       }
        // configure decoder, set sampleBuffer, etc. Note not all steps included, just setting the 
        // callback for the session and the decode. 
        if(status == noErr)
        {
            status = VTDecompressionSessionCreate(allocator: kCFAllocatorDefault, formatDescription: formatDesc!, decoderSpecification: nil, imageBufferAttributes: nil, outputCallback: &callback, decompressionSessionOut: &session);
        } else{
            rtn = ["image": [], "status": status];
            return result(rtn)
        }

        if(status == noErr) {
            status = VTDecompressSessionDecodeFrame(session!, sampleBuffer: sampleBuffer!, flags:               
                [._EnableTemporalProcessing], frameRefCon: frameRef, infoFlagsOut: nil)
        }

        // Return [UInt8] image and status code
    }
}

Answer 1

Solved by using the following override for the function:

func VTDecompressionSessionDecodeFrame(
    _ session: VTDecompressionSession,
    sampleBuffer: CMSampleBuffer,
    flags decodeFlags: VTDecodeFrameFlags,
    infoFlagsOut: UnsafeMutablePointer<VTDecodeInfoFlags>?,
    outputHandler: @escaping VTDecompressionOutputHandler
) -> OSStatus

There's also a way to pass self to the callback but haven't really looked into that.