IOS ML Kit face tracking does not work correctly

Question

The tracking Id of face detection is keeping change while the face is not moving, I use ML Kit in ios and I followed the documentation of google.

The documentation: https://developers.google.com/ml-kit/vision/face-detection/ios#performance_tips

Here is my code :

func captureOutput(  _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
         self.detectFaceMLKit(image: sampleBuffer)
}

private func detectFaceMLKit(image: CMSampleBuffer) {
        let image = VisionImage(buffer: image)
        image.orientation = imageOrientation(
            deviceOrientation: UIDevice.current.orientation,
            cameraPosition: cameraPosition)
               
        faceDetector?.process(image) { [self] faces, error in
          guard error == nil, let faces = faces, !faces.isEmpty else {
            //TODO : show error
            return
          }

            // Faces detected
            if faces.count == 1 {
                let face = faces[0]
                let frame = face.frame
                var rotX : CGFloat = 0
                var rotY : CGFloat = 0
                var faceTrackingId = 0
                if face.hasHeadEulerAngleX {
                    rotX = face.headEulerAngleX  // Head is rotated to the uptoward rotX degrees
                    print("rotx: \(rotX)")
                }
                if face.hasHeadEulerAngleY {
                    rotY = face.headEulerAngleY  // Head is rotated to the right rotY degrees
                    print("rotY: \(rotY)")
                }
                // If face tracking was enabled:
                if face.hasTrackingID {
                    faceTrackingId = face.trackingID
                    print("trackingId: \(faceTrackingId)")
                }
    }
}

    func imageOrientation(
        deviceOrientation: UIDeviceOrientation,
        cameraPosition: AVCaptureDevice.Position
    ) -> UIImage.Orientation {
        switch deviceOrientation {
        case .portrait:
            return cameraPosition == .front ? .leftMirrored : .right
        case .landscapeLeft:
            return cameraPosition == .front ? .downMirrored : .up
        case .portraitUpsideDown:
            return cameraPosition == .front ? .rightMirrored : .left
        case .landscapeRight:
            return cameraPosition == .front ? .upMirrored : .down
        case .faceDown, .faceUp, .unknown:
            return .up
        }
    }

Note: the tracking id works fine at the beginning but after around the id "10" it loses the face tracking and starts to give a multi id for the same face without moving the face in the camera.

An output sample after filtering the tracking id :

trackingId: 21 trackingId: 21 trackingId: 21 trackingId: 21 trackingId: 21 trackingId: 21 trackingId: 22 trackingId: 22 trackingId: 22 trackingId: 22 trackingId: 22 trackingId: 23 trackingId: 23 trackingId: 23 trackingId: 23 trackingId: 23 trackingId: 24 trackingId: 24 trackingId: 24 trackingId: 24 trackingId: 24

Answer 1

The problem was the imageOrientation, I set the orientation to portrait only in Xcode but rotating the image-based on UIDeviceOrientation which is wrong, fixing it by setting the imageOrientation to be fixed at .up position.

Edit : Also, make sure you don't override the output image orientation like this:

 private let videoDataOutput = AVCaptureVideoDataOutput()
 guard let connection = self.videoDataOutput.connection(with: AVMediaType.video),
            connection.isVideoOrientationSupported else { return }
 connection.videoOrientation = .portrait