What does the error "Exceeded maximum allowed number of Trackers" mean?

Question

I'm following this tutorial to implement object tracking on iOS 11. I'm able to track objects perfectly, until a certain point, then this error appears in the console.

Throws: Error Domain=com.apple.vis Code=9 "Internal error: Exceeded maximum allowed number of Trackers for a tracker type: VNObjectTrackerType" UserInfo={NSLocalizedDescription=Internal error: Exceeded maximum allowed number of Trackers for a tracker type: VNObjectTrackerType}

Am I using the API incorrectly, or perhaps Vision has trouble handling too many consecutive object tracking tasks? Curious if anyone has insight into why this is happening.

Answer 1

My problem with this was that I had a function that called perform... on the same VNSequenceRequestHandler that the tracking was also calling perform on, because of that I was processing too many try self.visionSequenceHandler.perform(trackRequests, on: ciimage) concurrently. Make sure the VNSequenceRequestHandler is not getting hit at the same time by multiple performs....

Answer 2

To track the rectangle you feed consequent observations to the same VNSequenceRequestHandler instance, say, handler. When the rectangle is lost, i.e. the new observation is nil in your handler function / callback, or you are getting some other tracking error, just re-instantiate the handler and continue, e.g. (sample code to show the idea):

private var handler = VNSequenceRequestHandler()

// <...>

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    guard
        let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer),
        let lastObservation = self.lastObservation
    else { 
        self.handler = VNSequenceRequestHandler()
        return
    }

    let request = VNTrackObjectRequest(detectedObjectObservation: lastObservation, completionHandler: self.handleVisionRequestUpdate)
    request.trackingLevel = .accurate

    do {
        try self.handler.perform([request], on: pixelBuffer)
    } catch {
        print("Throws: \(error)")
    }
}

Note that handler is var, not a constant.

Also, you may re-instantiate the handler in actual handler function (like func handleVisionRequestUpdate(_ request: VNRequest, error: Error?)) in case new observation object is invalid.

Answer 3

You have to release the tracker of the request whenever the tracking observation is deemed of no use anymore with isLastFrame property.

        guard let processedRequest = processedRequest as? VNTrackObjectRequest
        else { continue }
        
        guard let results = processedRequest.results,
              let observation = results.first as? VNDetectedObjectObservation,
              observation.confidence > 0.1
        else {
            // Tells Vision that we want to release the tracker reserved for this request, freeing resources
            processedRequest.isLastFrame = true
            
            // Feed the request again into Vision, effectively taking into account the new isLastFrame value
            do {
                try trackingRequestHandler.perform([processedRequest], on: frame, orientation: .up)
            } catch {
                print(error)
            }
            continue
        }

It's important to perform the request again after setting isLastFrame property so that it can be read by Vision.

Answer 4

It appears that you hit the limit on the number of trackers that can be active in the system. First thing to note is that a new tracker is created every time a new observation, with new -uuid property is used. You should be recycling the initial observation you use when you started the tracker, until you no longer want to use it, by feeding what you got from “results” for time T into the subsequent request you make for time T+1. When you no longer want to use that tracker (maybe the confidence score gets too low), there is a “lastFrame” property that can be set, which lets the Vision framework know that you are done with that tracker. Trackers also get released when the sequence request handler is released.

Answer 5

To avoid "Exceeded maximum allowed number of Trackers" error, you have to stop trackers for all the objects you were tracking. You have to set isLastFrame to true and perform this request on a handler once again.

extension VNTrackingRequest {
        func completeTracking(
            with handler: VNSequenceRequestHandler,
            on sampleBuffer: CMSampleBuffer
        ) {
            isLastFrame = true
            try? handler.perform([self], on: sampleBuffer)
        }
    }

Here is a processing example. If there is no observation, ObjectTracker tries to locate a new object. However if there is an object detected already (observation), it will track it's location.

class ObjectTracker: CameraViewControllerOutputDelegate {

var observation: VNDetectedObjectObservation?

private lazy var objectDetectRequest: VNCoreMLRequest = newObjectDetectRequest()
private var objectTrackingRequest: VNTrackObjectRequest?

private func newObjectDetectRequest() -> VNCoreMLRequest {
    do {
        let mlModel = try YourMLModel(configuration: .init()).model
        let model = try VNCoreMLModel(for: mlModel)
        let request = VNCoreMLRequest(model: model)
        return request
    } catch {
        fatalError("Failed to load ML Model. Error: \(error)")
    }
}

private func newObjectTrackingRequest(
    for observation: VNDetectedObjectObservation
) -> VNTrackObjectRequest {
    let request = VNTrackObjectRequest(detectedObjectObservation: observation)
    request.trackingLevel = .fast
    return request
}
    
func cameraViewController(
    _ controller: CameraViewController,
    didReceiveBuffer buffer: CMSampleBuffer,
    orientation: CGImagePropertyOrientation
) {
    if let oldObservation = observation {
        objectTrackingRequest = objectTrackingRequest ?? newObjectTrackingRequest(for: oldObservation)
        let visionHandler = VNSequenceRequestHandler()
        try? visionHandler.perform([objectTrackingRequest!], on: buffer, orientation: orientation)

        if let newObservation = objectTrackingRequest?.results?.first as? VNDetectedObjectObservation {
            observation = newObservation
            objectTrackingRequest?.inputObservation = newObservation
        } else {
            observation = nil
            objectTrackingRequest?.completeTracking(with: visionHandler, on: buffer)
            objectTrackingRequest = nil
        }
        
    } else {
       objectDetectRequest = newObjectDetectRequest()
        let visionHandler = VNImageRequestHandler(cmSampleBuffer: buffer, orientation: orientation, options: [:] )
        try? visionHandler.perform([objectDetectRequest])
        
        if let newObservation = objectDetectRequest.results?.first as? VNRecognizedObjectObservation {
            observation = newObservation
        }
    }
}
}