Save depth images from TrueDepth camera

Question

I am trying to save depth images from the iPhoneX TrueDepth camera. Using the AVCamPhotoFilter sample code, I am able to view the depth, converted to grayscale format, on the screen of the phone in real-time. I cannot figure out how to save the sequence of depth images in the raw (16 bits or more) format.

I have depthData which is an instance of AVDepthData. One of its members is depthDataMap which is an instance of CVPixelBuffer and image format type kCVPixelFormatType_DisparityFloat16. Is there a way to save it to the phone to transfer for offline manipulation?

Answer 1

There's no standard video format for "raw" depth/disparity maps, which might have something to do with AVCapture not really offering a way to record it.

You have a couple of options worth investigating here:

1. Convert depth maps to grayscale textures (which you can do using the code in the AVCamPhotoFilter sample code), then pass those textures to AVAssetWriter to produce a grayscale video. Depending on the video format and grayscale conversion method you choose, other software you write for reading the video might be able to recover depth/disparity info with sufficient precision for your purposes from the grayscale frames.

2. Anytime you have a CVPixelBuffer, you can get at the data yourself and do whatever you want with it. Use CVPixelBufferLockBaseAddress (with the readOnly flag) to make sure the content won't change while you read it, then copy data from the pointer CVPixelBufferGetBaseAddress provides to wherever you want. (Use other pixel buffer functions to see how many bytes to copy, and unlock the buffer when you're done.)

   Watch out, though: if you spend too much time copying from buffers, or otherwise retain them, they won't get deallocated as new buffers come in from the capture system, and your capture session will hang. (All told, it's unclear without testing whether a device has the memory & I/O bandwidth for much recording this way.)

Answer 2

You can use Compression library to create a zip file with the raw CVPixelBuffer data. Few problems with this solution.

1. It's a lot of data and zip is not a good compression. (the compressed file is 20 times bigger than 32bits per frame video with the same number of frames).
2. Apple's Compression library creates a file which standard zip program does't open. I use zlib in C code to read it and use inflateInit2(&strm, -15); to make it work.
3. You'll need to do some work to export the file out of your application

Here is my code (which I limited to 250 frames since it hold it in RAM but you can flush to disk if needed more frames):

//  DepthCapture.swift
//  AVCamPhotoFilter
//
//  Created by Eyal Fink on 07/04/2018.
//  Copyright © 2018 Resonai. All rights reserved.
//
// Capture the depth pixelBuffer into a compress file.
// This is very hacky and there are lots of TODOs but instead we need to replace
// it with a much better compression (video compression)....

import AVFoundation
import Foundation
import Compression


class DepthCapture {
    let kErrorDomain = "DepthCapture"
    let maxNumberOfFrame = 250
    lazy var bufferSize = 640 * 480 * 2 * maxNumberOfFrame  // maxNumberOfFrame frames
    var dstBuffer: UnsafeMutablePointer<UInt8>?
    var frameCount: Int64 = 0
    var outputURL: URL?
    var compresserPtr: UnsafeMutablePointer<compression_stream>?
    var file: FileHandle?

    // All operations handling the compresser oobjects are done on the
    // porcessingQ so they will happen sequentially
    var processingQ = DispatchQueue(label: "compression",
                                    qos: .userInteractive)


    func reset() {
        frameCount = 0
        outputURL = nil
        if self.compresserPtr != nil {
            //free(compresserPtr!.pointee.dst_ptr)
            compression_stream_destroy(self.compresserPtr!)
            self.compresserPtr = nil
        }
        if self.file != nil {
            self.file!.closeFile()
            self.file = nil
        }
    }
    func prepareForRecording() {
        reset()
        // Create the output zip file, remove old one if exists
        let documentsPath = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as NSString
        self.outputURL = URL(fileURLWithPath: documentsPath.appendingPathComponent("Depth"))
        FileManager.default.createFile(atPath: self.outputURL!.path, contents: nil, attributes: nil)
        self.file = FileHandle(forUpdatingAtPath: self.outputURL!.path)
        if self.file == nil {
            NSLog("Cannot create file at: \(self.outputURL!.path)")
            return
        }

        // Init the compression object
        compresserPtr = UnsafeMutablePointer<compression_stream>.allocate(capacity: 1)
        compression_stream_init(compresserPtr!, COMPRESSION_STREAM_ENCODE, COMPRESSION_ZLIB)
        dstBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: bufferSize)
        compresserPtr!.pointee.dst_ptr = dstBuffer!
        //defer { free(bufferPtr) }
        compresserPtr!.pointee.dst_size = bufferSize


    }
    func flush() {
        //let data = Data(bytesNoCopy: compresserPtr!.pointee.dst_ptr, count: bufferSize, deallocator: .none)
        let nBytes = bufferSize - compresserPtr!.pointee.dst_size
        print("Writing \(nBytes)")
        let data = Data(bytesNoCopy: dstBuffer!, count: nBytes, deallocator: .none)
        self.file?.write(data)
    }

    func startRecording() throws {
        processingQ.async {
            self.prepareForRecording()
        }
    }
    func addPixelBuffers(pixelBuffer: CVPixelBuffer) {
        processingQ.async {
            if self.frameCount >= self.maxNumberOfFrame {
                // TODO now!! flush when needed!!!
                print("MAXED OUT")
                return
            }

            CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
            let add : UnsafeMutableRawPointer = CVPixelBufferGetBaseAddress(pixelBuffer)!
            self.compresserPtr!.pointee.src_ptr = UnsafePointer<UInt8>(add.assumingMemoryBound(to: UInt8.self))
            let height = CVPixelBufferGetHeight(pixelBuffer)
            self.compresserPtr!.pointee.src_size = CVPixelBufferGetBytesPerRow(pixelBuffer) * height
            let flags = Int32(0)
            let compression_status = compression_stream_process(self.compresserPtr!, flags)
            if compression_status != COMPRESSION_STATUS_OK {
                NSLog("Buffer compression retured: \(compression_status)")
                return
            }
            if self.compresserPtr!.pointee.src_size != 0 {
                NSLog("Compression lib didn't eat all data: \(compression_status)")
                return
            }
            CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)
            // TODO(eyal): flush when needed!!!
            self.frameCount += 1
            print("handled \(self.frameCount) buffers")
        }
    }
    func finishRecording(success: @escaping ((URL) -> Void)) throws {
        processingQ.async {
            let flags = Int32(COMPRESSION_STREAM_FINALIZE.rawValue)
            self.compresserPtr!.pointee.src_size = 0
            //compresserPtr!.pointee.src_ptr = UnsafePointer<UInt8>(0)
            let compression_status = compression_stream_process(self.compresserPtr!, flags)
            if compression_status != COMPRESSION_STATUS_END {
                NSLog("ERROR: Finish failed. compression retured: \(compression_status)")
                return
            }
            self.flush()
            DispatchQueue.main.sync {
                success(self.outputURL!)
            }
            self.reset()
        }
    }
}