Android CameraX: Show image analysis output bitmap back to PreviewView

Question

I am using the CameraX image analysis API to get each frame of a camera feed.

I then process the image proxy through a ML model which transforms the image frame. The output of the ML pipeline is a bitmap of the camera frame with a new style.

How can I display the inference output bitmap back to the PreviewView in a Jetpack compose app? This is on a live camera feed.

Answer 1

Please check out the OverlayEffect API. You need to create a OverlayEffect with a certain queue depth. The depth depends on how fast your ML model is. Then upon your ML model returns the Bitmap, release the preview frame with OverlayEffect#drawFrameAsync. Untested sample code:

val effect = OverlayEffect(PREVIEW | VIDEO_CAPTURE, 8, handler) {}
effect.setOnDrawListener { frame ->
  val canvas = frame.getOverlayCanvas()
  canvas.drawBitmap(...)
  return true
}

imageAnalysis.setAnalyzer(executor)  { imageProxy ->
  val bitmap = getBitmapWithMlModel(imageProxy)
  // release the frame from the queue
  effect.drawFrameAsync(imageProxy.imageInfo.timestamp)
  // The draw the Bitmap in the listener above
}

However, I should point out that analyzing the ImageAnalysis image and generating a Bitmap happens on CPU which can be slow. The ideal way is analyze and draw the image with GPU, e.g. using OpenGL.