Convert WebRTC i420frame to YUV_420_888 and NV21

94 views Asked by At

I am using libjingle_peerconnection.jar for video calling based on this project and everything is fine.

Now I need to detect face in real time and also detect smile. I want to use ML Kit for this. Therefore, I need to access the byte buffer of each frame.

As Google says:

If you use the Camera2 API, capture images in ImageFormat.YUV_420_888 format. If you use the older Camera API, capture images in ImageFormat.NV21 format.

To access the frame, I copied and pasted the SurfaceViewRenderer source into my project and used it. In this class, there is a method called renderFrame that takes a org.webrtc.VideoRenderer.I420Frame as a parameter.

There are two problems.

  1. When the camera is used, the yuvPlanes value of the frame is null.
  2. When remote is used, the yuvPlanes value is not null, but I can't convert it to YUV_420_888 and NV21 frame. How should I do this?

I420Frame class properties:

    public static class I420Frame {
        public final int width;
        public final int height;
        public final int[] yuvStrides;
        public ByteBuffer[] yuvPlanes;
        public final boolean yuvFrame;
        public final float[] samplingMatrix;
        public int textureId;
        private long nativeFramePointer;
        public int rotationDegree;
    }
0

There are 0 answers