I am using libjingle_peerconnection.jar
for video calling based on this project and everything is fine.
Now I need to detect face in real time and also detect smile. I want to use ML Kit for this. Therefore, I need to access the byte buffer of each frame.
As Google says:
If you use the Camera2 API, capture images in ImageFormat.YUV_420_888 format. If you use the older Camera API, capture images in ImageFormat.NV21 format.
To access the frame, I copied and pasted the SurfaceViewRenderer
source into my project and used it. In this class, there is a method called renderFrame
that takes a org.webrtc.VideoRenderer.I420Frame
as a parameter.
There are two problems.
- When the camera is used, the
yuvPlanes
value of the frame isnull
. - When remote is used, the
yuvPlanes
value is not null, but I can't convert it toYUV_420_888
andNV21
frame. How should I do this?
I420Frame
class properties:
public static class I420Frame {
public final int width;
public final int height;
public final int[] yuvStrides;
public ByteBuffer[] yuvPlanes;
public final boolean yuvFrame;
public final float[] samplingMatrix;
public int textureId;
private long nativeFramePointer;
public int rotationDegree;
}