I am using libjingle_peerconnection.jar for video calling based on this project and everything is fine.
Now I need to detect face in real time and also detect smile. I want to use ML Kit for this. Therefore, I need to access the byte buffer of each frame.
As Google says:
If you use the Camera2 API, capture images in ImageFormat.YUV_420_888 format. If you use the older Camera API, capture images in ImageFormat.NV21 format.
To access the frame, I copied and pasted the SurfaceViewRenderer source into my project and used it. In this class, there is a method called renderFrame that takes a org.webrtc.VideoRenderer.I420Frame as a parameter.
There are two problems.
- When the camera is used, the
yuvPlanesvalue of the frame isnull. - When remote is used, the
yuvPlanesvalue is not null, but I can't convert it toYUV_420_888andNV21frame. How should I do this?
I420Frame class properties:
public static class I420Frame {
public final int width;
public final int height;
public final int[] yuvStrides;
public ByteBuffer[] yuvPlanes;
public final boolean yuvFrame;
public final float[] samplingMatrix;
public int textureId;
private long nativeFramePointer;
public int rotationDegree;
}