Drawing ARCore Camera Feed onto Unity Texture

Question

I’m trying to display the ARCore camera feed onto a texture I’ve created in Unity, but I can’t see to figure out how to get it to work correctly.

I’ve created my texture in Unity like this:

Texture2D unityTexture = new Texture2D(
        480,
        640,
        TextureFormat.RGBA32,
        false);
 unityTexture.filterMode = FilterMode.Point;
 unityTexture.Apply();

And I pass over my texture id into Java using the value returned by Texture2D#GetNativeTexturePtr().

In my Java, I try to set the texture as follows:

  // Called from Unity layer when camera texture is created.
  public void setUnityCameraTexture(long texId, int width, int height) {
    int textureId = (int) textId;
    mSession.setCameraTextureName(textureId);
    mSession.setDisplayGeometry(width, height);
  }

...

  // Called on each Unity MonoBehaviour.Update().
  public void renderFrame() {
    try {
      mSession.update();
    } catch (CameraException e) {
      Log.w(TAG, "Error when updating session", e);
    }
  }

With this approach, the texture is not rendered to the screen and I received the following error logs:

E GLConsumer: [SurfaceTexture-1337-7436-0] bindTextureImage: error binding external image: 0x502
E tango_camera_native: Couldn't get image exposure/skew for timestamp offset!
E ndk_camera: metadata has: 780757.577090
E ndk_camera: metadata has: 780757.890735
E ndk_camera: metadata has: 780757.924221
E ndk_camera: metadata has: 780757.957402
E ndk_camera: metadata has: 780757.990704
E ndk_camera: metadata has: 780758.024007
E ndk_camera: metadata has: 780758.057341
E ndk_camera: metadata has: 780758.090735
E ndk_camera: metadata has: 780758.124007
E ndk_camera: metadata has: 780758.157341
E ndk_camera: metadata has: 780758.190613
E ndk_camera: metadata has: 780758.223977
E ndk_camera: metadata has: 780758.257432
E ndk_camera: metadata has: 780758.290643
E ndk_camera: metadata has: 780758.323946
E ndk_camera: metadata has: 780758.349104
W tango_camera_native_jni: jint Java_com_google_tango_jni_TangoCameraNative_UpdateTextureExternalOes(JNIEnv*, jobject, int, int, jdoubleArray): Got a negative timestamp -0.000000 from camera with id 0.
I Unity   : AndroidJavaException: java.lang.RuntimeException: Error during updateTexImage (see logcat for details)
I Unity   : java.lang.RuntimeException: Error during updateTexImage (see logcat for details)
I Unity   :     at android.graphics.SurfaceTexture.nativeUpdateTexImage(Native Method)
I Unity   :     at android.graphics.SurfaceTexture.updateTexImage(SurfaceTexture.java:244)
I Unity   :     at com.google.tango.jni.TangoCameraNative.UpdateTextureExternalOes(Native Method)
I Unity   :     at com.google.tango.jni.TangoCameraNative.updateTextureExternalOes(TangoCameraNative.java:222)
I Unity   :     at com.google.tango.loader.ITangoCameraNative$Stub.onTransact(ITangoCameraNative.java:162)
I Unity   :     at android.os.Binder.transact(Binder.java:499)
I Unity   :     at com.google.tango.loader.ITangoCameraNative$Stub$Proxy.updateTextureExternalOes(ITangoCameraNative.java:436)
I Unity   :     at com.google.atap.tangoservice.TangoCameraNativeLoader.updateTextureExternalOes(TangoCameraNativeLoader.java:178)
I Unity   :     at com.google.atap.tangoservice.Tango.updateTextureExternalOes(Tango.java:512)
I Unity   :     at com.google.ar.core.TangoWrapper.updateTextureExternalOes(Tan

Not sure if I've just setup my texture incorrectly, but any help would be greatly appreciated!

Answer 1

Figured it out. I wasn’t setting up my texture properly. ARCore documentation states that the texture must be created withglGenTextures(int, int[], int) and bound to the GL_TEXTURE_EXTERNAL_OES target. Instead of allowing Unity to create the texture, I'm now creating the texture at the Java layer, binding it to GL_TEXTURE_EXTERNAL_OES, then passing the texture ID back to Unity.

My Java now looks something like:


// Called on app resume.
public void resume() {
  mSession.resume(mConfig);
  mSession.setCameraTextureName(mTextureName[0]);
}

// Called on each Unity MonoBehaviour.Update().
public void renderFrame() {
  try {
    mSession.update();
  } catch (CameraException e) {
    Log.w(TAG, "Error when updating session", e);
  }
}

public int getTextureName() {
  mTextureName = new int[1];
  GLES20.glGenTextures(1, mTextureName, 0);
  GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, mTextureName[0]);
  return mTextureName[0];
}

And in my Unity scripts:

// Called on Unity Start()
public Texture2D GetCameraTexture() {
  IntPtr textureId = GetCameraTextureId();
  Texture2D cameraTexture = Texture2D.CreateExternalTexture(
      1080,
      1920,
      TextureFormat.RGBA32,
      false,
      false,
      GetTextureId());
  return cameraTexture;
}

private IntPtr GetCameraTextureId() {
  // Calls into Java getTextureName method.
}

In addition to those changes, I also had to create a custom Unity GLSL shader which accessed the texture using the samplerExternalOES sampler, and use that to display the texture on one of my GameObjects. This was also specified as a requirement in the documentation.