Android camera2.params.face rectangle placement on canvas

Question

I'm trying to implement face detection in my camera preview. I followed the Android reference pages to implement a custom camera preview in a TextureView, placed in a FrameLayout. Also in this FrameLayout is a SurfaceView with a clear background (overlapping the camera preview). My app draws the Rect that is recognized by the first CaptureResult.STATISTICS_FACES face's bounds dynamically to the SurfaceView's canvas, every time the camera preview is updated (once per frame). My app assumes only one face will need to be recognized.

My issue arises when I draw the rectangle. I get the rectangle in the correct place if I keep my face in the center of the camera view, but when I move my head upward the rectangle moves to the right, and when I move my head to the right, the rectangle moves down. It's as if the canvas needs to be rotated by -90, but this doesn't work for me (Explained below code).

in my activity's onCreate():

//face recognition
rectangleView = (SurfaceView) findViewById(R.id.rectangleView);
rectangleView.setZOrderOnTop(true);
rectangleView.getHolder().setFormat(
    PixelFormat.TRANSPARENT); //remove black background from view
purplePaint = new Paint();
purplePaint.setColor(Color.rgb(175,0,255));
purplePaint.setStyle(Paint.Style.STROKE);

in TextureView.SurfaceTextureListener.onSurfaceTextureAvailable()(in the try{} block that encapsulates camera.open() :

Rect cameraBounds = cameraCharacteristics.get(
    CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE);
cameraWidth = cameraBounds.right;
cameraHeight = cameraBounds.bottom;

in the same listener within onSurfaceTextureUpdated() :

if (detectedFace != null && rectangleFace.height() > 0) {
        Canvas currentCanvas = rectangleView.getHolder().lockCanvas();
        if (currentCanvas != null) {
            currentCanvas.drawColor(Color.TRANSPARENT, PorterDuff.Mode.CLEAR);
            int canvasWidth = currentCanvas.getWidth();
            int canvasHeight = currentCanvas.getHeight();
            int l = rectangleFace.right;
            int t = rectangleFace.bottom;
            int r = rectangleFace.left;
            int b = rectangleFace.top;
            int left = (canvasWidth*l)/cameraWidth;
            int top  = (canvasHeight*t)/cameraHeight;
            int right = (canvasWidth*r)/cameraWidth;
            int bottom = (canvasHeight*b)/cameraHeight;

            currentCanvas.drawRect(left, top, right, bottom, purplePaint);
        }
        rectangleView.getHolder().unlockCanvasAndPost(currentCanvas);
    }

method onCaptureCompleted in CameraCaptureSession.CameraCallback called by CameraCaptureSession.setRepeatingRequest() looper:

//May need better face recognition sdk or api
Face[] faces = result.get(CaptureResult.STATISTICS_FACES);

if (faces.length > 0)
{
    detectedFace = faces[0];
    rectangleFace = detectedFace.getBounds();
}

All variables are instantiated outside of the functions.

In case you can't quite understand my question or need additional information, a similar question is posted here:

How can i handle the rotation issue with Preview & FaceDetection

However, unlike the above poster, I couldn't even get my canvas to show the rectangle after rotating my canvas, so that can't be the solution.

I tried to rotate my points by -90 degrees using x=-y, y=x (left=-top, top=left), and it doesn't do the trick either. I feel like some kind of function needs to be applied to the points but I don't know how to go about it.

Any ideas on how to fix this?

Answer 1

For future reference, this is the solution I ended up with:

set a class/Activity variable called orientation_offset :

orientation_offset = cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION);

This is the angle that the camera sensor's view (or rectangle for face detection) needs to be rotated to be viewed correctly.

Then, I changed onSurfaceTextureUpdated() :

Canvas currentCanvas = rectangleView.getHolder().lockCanvas();
    if (currentCanvas != null) {

        currentCanvas.drawColor(Color.TRANSPARENT, PorterDuff.Mode.CLEAR);

        if (detectedFace != null && rectangleFace.height() > 0) {

            int canvasWidth = currentCanvas.getWidth();
            int canvasHeight = currentCanvas.getHeight();
            int faceWidthOffset = rectangleFace.width()/8;
            int faceHeightOffset = rectangleFace.height()/8;

            currentCanvas.save();
            currentCanvas.rotate(360 - orientation_offset, canvasWidth / 2, 
                canvasHeight / 2);

            int l = rectangleFace.right;
            int t = rectangleFace.bottom;
            int r = rectangleFace.left;
            int b = rectangleFace.top;
            int left = (canvasWidth - (canvasWidth*l)/cameraWidth)-(faceWidthOffset);
            int top  = (canvasHeight*t)/cameraHeight - (faceHeightOffset);
            int right = (canvasWidth - (canvasWidth*r)/cameraWidth) + (faceWidthOffset);
            int bottom = (canvasHeight*b)/cameraHeight + (faceHeightOffset);

            currentCanvas.drawRect(left, top, right, bottom, purplePaint);
            currentCanvas.restore();
        }
    }
    rectangleView.getHolder().unlockCanvasAndPost(currentCanvas);

I'll leave the question open in case somebody else has a solution to offer.