Accessing Image Data Bytes in ARCore

Question

I've created an ARCore Session and attached an OpenGL texture id through the Session#setCameraTextureName method to display my camera data. I'd like to have access to the camera image data bytes displayed on the texture.

ARKit and Tango provide access to the image bytes for each frame but there doesn't seem to be anything that easily provides that in the ARCore API.

Is there any other way I can access the image bytes when using ARCore?

Answer 1

As of ARCore v1.1.0, there is an API to access the image bytes for the current frame:

https://developers.google.com/ar/reference/java/com/google/ar/core/Frame.html#acquireCameraImage()

Answer 2

For the time being, your best bet for accessing image data is probably drawing the texture to a renderbuffer and using glReadPixels into a persistent-mapped pixel unpack buffer. Use a fence sync to detect when the glReadPixels is complete.

Another option is to use a compute shader and write directly to a persistent-mapped SSBO. (Disregard persistent-mapped suggestion. I thought EXT_buffer_storage had broader support)

The later is possibly fewer copies (the renderbuffer pixels may still hit DRAM even if you invalidate it after the glReadPixels), but it's also a less-common code path and incurs render/compute changeovers so I don't have intuition about which approach would be more efficient.

Answer 3

Maybe that could help you I wanted to obtain the camera view in a bitmap form. I have tested on Samsung s8.

    int w=1080;
    int h = 2220;
    int b[]=new int[w*(0+h)];
    int bt[]=new int[w*h];
    IntBuffer ib = IntBuffer.wrap(b);
    ib.position(0);
    GLES20.glReadPixels(0, 0, w, h, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, ib);

    for(int i=0, k=0; i<h; i++, k++)
    {//remember, that OpenGL bitmap is incompatible with Android bitmap
        //and so, some correction need.
        for(int j=0; j<w; j++)
        {
            int pix=b[i*w+j];
            int pb=(pix>>16)&0xff;
            int pr=(pix<<16)&0x00ff0000;
            int pix1=(pix&0xff00ff00) | pr | pb;
            bt[(h-k-1)*w+j]=pix1;
        }
    }

    sb=Bitmap.createBitmap(bt, w, h, Bitmap.Config.ARGB_8888);