VNTrackRectangleRequest internal error

Question

I'm trying to get a simple rectangle tracking controller going, and I can get rectangle detection going just fine, but the tracking request always ends up failing for a reason I can't quite find.

Sometimes the tracking request will fire it's callback a few times before failing, other times it fails immediately before a single callback occurs. I feel it's something to do with how I submit the requests but I can't get to the bottom of it.

Here's the code for the view controller

class TestController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate {

    // Video capture
    private var videoSession = AVCaptureSession()
    private var videoLayer: AVCaptureVideoPreviewLayer!

    // Detection
    private var detectionRequest: VNDetectRectanglesRequest?
    private let sequenceHandler = VNSequenceRequestHandler()

    // Tracking
    private var trackingRequest: VNTrackRectangleRequest?
    private var shape: Detection?
    private var pixelBuffer: CVImageBuffer?




    // MARK: Setup
    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)

        startVideoFeed()
    }

    override func viewDidLayoutSubviews() {
        videoLayer.frame = view.layer.bounds
    }

    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        startDetectingRectangles()
    }



    private func startDetectingRectangles() {
        let request = VNDetectRectanglesRequest(completionHandler: didDetectRectangle)
        request.maximumObservations = 1
        request.minimumSize = 0.07
        request.minimumConfidence = 0.9
        request.minimumAspectRatio = 0.5
        request.maximumAspectRatio = 2
        request.quadratureTolerance = 10
        detectionRequest = request
    }

    private func didDetectRectangle(request: VNRequest, error: Error?) {

        // Fetch results of the correct type
        guard let observations = request.results, observations.count > 0 else { return }
        let results = observations.map { $0 as? VNRectangleObservation }

        for case let rectangle? in results {
            detectionRequest = nil

            let request = VNTrackRectangleRequest(rectangleObservation: rectangle, completionHandler: didTrackRectangle)
            trackingRequest = request
        }
    }


    private func didTrackRectangle(request: VNRequest, error: Error?) {

        // Fetch results of the correct type
        guard let observation = request.results?.first as? VNRectangleObservation else { return }

        // Create or update UI
    }




    // Start capturing video frames
    private func startVideoFeed() {

        // Session config
        videoSession.sessionPreset = .photo

        // Create device and input to device
        guard
            let captureDevice = AVCaptureDevice.default(for: .video),
            let deviceInput = try? AVCaptureDeviceInput(device: captureDevice)
            else {
                fatalError("Error setting up capture device.")
        }

        // Setup device output
        let deviceOutput = AVCaptureVideoDataOutput()
        deviceOutput.setSampleBufferDelegate(self, queue: DispatchQueue.global(qos: .default))

        // Set input and output
        videoSession.addInput(deviceInput)
        videoSession.addOutput(deviceOutput)

        // Setup video display layer
        videoLayer = AVCaptureVideoPreviewLayer(session: videoSession)
        videoLayer.frame = view.bounds
        videoLayer.videoGravity = .resizeAspectFill
        view.layer.addSublayer(videoLayer)

        videoSession.startRunning()
    }

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

        pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)

        guard let pixelBuffer = pixelBuffer else { return }
        do {

            var requests: [VNRequest] = []

            if let detectionRequest = detectionRequest {
                requests.append(detectionRequest) }
            if let trackingRequest = trackingRequest {
                requests.append(trackingRequest) }

            try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right)
        } catch {
            print(error)
        }
    }
}

The error is printing:

Error Domain=com.apple.vis Code=9 "Internal error: Tracking of one of the corners failed, confidence = 0.000000; threshold = 0.650000" UserInfo={NSLocalizedDescription=Internal error: Tracking of one of the corners failed, confidence = 0.000000; threshold = 0.650000}

This is being output every frame after the tracking request first fails, and the request never recovers itself. It's the try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right) line that is throwing the error (which I then print).

I've looked at the object observation examples dotted around the web and they seem to require a new tracking request to be made each frame to track the object. I've tried creating a new VNTrackRectangleRequest in the didTrackRectangle function so each frame has a new request, but I get the same issue.

Any help regarding this is really appreciated.

Answer 1

I was seeing this error every frame, and to fix it I found that I needed to call performRequests:onCVPixelBuffer:orientation:error: with the correct orientation.

(For portrait orientation with the rear-facing camera, I believe the orientation is kCGImagePropertyOrientationRight).

From the documentation:

For each such request, call the sequence request handler’s performRequests:onCVPixelBuffer:orientation:error: method, making sure to pass in the video reader’s orientation to ensure upright tracking.

Answer 2

I was able to run your code without any issues, at least when tracking a clearly visible rectangle.

You should probably create a new VNSequenceRequestHandler when the tracking request fails. It seems like when a tracking request loses track of an object for too long, it won't be able to recover and start tracking it again. This also means they never get released and will keep throwing errors. If you continue to add new tracking requests to your request handler, you'll quickly be over the limit for simultaneous tracking requests.

If you're able to anticipate the end of a tracking sequence, you can set the isLastFrame property of your tracking request to true, and it will be released to the pool of available trackers when the current frame finishes processing.