Obtain tango device main camera image in unity

Question

In my project I am capturing point cloud and based on that I create a binary mask, which I want to apply to captured image from Tango tablet main camera, so that the final image is a cutout of the detected points of point cloud.

I tried to use the tango examples for getting the camera image so that further postprocessing can be made, but I had no success. (temporary hack includes a camera to render texture and then applying a masking shader to the render)

What is the easiest method to obtain the tango main camera image in a Unity?

Answer 1

Take a look at the VideoOverlayProvider.cs file, there are two static methods SetCallback(..) which sound like what you are looking for:

    /// Connect a callback to a camera for access to the pixels.
    /// 
    /// This is not recommended for display but for applications requiring access to the
    /// <code>HAL_PIXEL_FORMAT_YV12</code> pixel data.  The camera is selected via TangoCameraId.  Currently only 
    /// <code>TANGO_CAMERA_COLOR</code> and <code>TANGO_CAMERA_FISHEYE</code> are supported.
    /// 
    /// The <i>onImageAvailable</i> callback will be called when a new frame is available from the camera. The
    /// Enable Video Overlay option must be enabled for this to succeed.
    ///
    /// Note: The first scan-line of the color image is reserved for metadata instead of image pixels.

You can get access to both the TANGO_CAMERA_COLOR camera and the TANGO_CAMERA_FISHEYE camera frames