How to fix video & audio out of sync issue on android Video Recording using Camera api 2?

Question

I am implementing video recording application using Camera2 api. I've used Google samples for video recording. However, there's an audio, video out of sync issue on some devices, such as Samsung J5, J6. I have changed MediaRecorder.AudioEncoder, MediaRecorder.VideoEncoder, VideoEncodingBitrate, but it could not help me. How to handle audio, video sync issue?

Answer 1

I've found a solution from this article. It mightn't be the best solution but it works. To solve the out of sync issue, mp4parserlibrary is utilized. First, video recording workflow is the same as just normal video recording, but , there's an extra step for the problematic devices. I will below provide my answer. First step of video recording is to prepare MediaRecorder, to shorten my answer I'll omit some steps.

 private void setupMediaRecorder(){
    mMediaRecorder = new MediaRecorder();

    mMediaRecorder.setAudioSource(MediaRecorder.AudioSource.DEFAULT);
    mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.SURFACE);
    mMediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);

    mMediaRecorder.setOutputFile(mVideoFilePath);
    mMediaRecorder.setVideoSize(mVideoSize.getWidth(), mVideoSize.getHeight());

    CamcorderProfile profile = CamcorderProfile.get(CamcorderProfile.QUALITY_480P);
    mMediaRecorder.setVideoFrameRate(profile.videoFrameRate);
    mMediaRecorder.setVideoSize(profile.videoFrameWidth, profile.videoFrameHeight);
    mMediaRecorder.setVideoEncodingBitRate(profile.videoBitRate);
    mMediaRecorder.setVideoEncoder(MediaRecorder.VideoEncoder.H264);
    mMediaRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);

    mMediaRecorder.setAudioChannels(2);
    mMediaRecorder.setAudioEncodingBitRate(profile.audioBitRate);
    mMediaRecorder.setAudioSamplingRate(profile.audioSampleRate);

    int rotation = ((WindowManager) mContext.getSystemService(Context.WINDOW_SERVICE)).getDefaultDisplay().getRotation();

    switch (getRotation()) {
        case SENSOR_ORIENTATION_DEFAULT_DEGREES:

            mMediaRecorder.setOrientationHint(ORIENTATIONS.get(rotation));
            break;
        case SENSOR_ORIENTATION_INVERSE_DEGREES:

            mMediaRecorder.setOrientationHint(INVERSE_ORIENTATIONS.get(rotation));
            break;
        case SENSOR_ORIENTATION_DEFAULT_LAND_DEGREES:

            mMediaRecorder.setOrientationHint((ORIENTATIONS.get(rotation)+270)%360);
            break;
        case SENSOR_ORIENTATION_INVERSE_LAND_DEGREES:

            mMediaRecorder.setOrientationHint((INVERSE_ORIENTATIONS.get(rotation)+270)%360);
            break;
    }
    try{
        mMediaRecorder.prepare();
    }catch (IllegalStateException | IOException exc){
        exc.printStackTrace();
    }
}

To stop video recording

public void stopVideo(){
    //Stop recording
    try {
       mMediaRecorder.stop();
       mMediaRecorder.release();
       parseVideo(mVideoFilePath);
    }catch (RuntimeException e){
    e.printStackTrace();
   }
   closePreviewSession();
   createCameraPreviewSession();
   if (mListener!=null){
    mListener.onPrepareRecorder();
   }
}

The last and important step is to call parseVideo function

 private String parseVideo(String mFilePath) {
    try {
        DataSource channel = new FileDataSourceImpl(mFilePath);
        IsoFile isoFile = new IsoFile(channel);
        List<TrackBox> trackBoxes = isoFile.getMovieBox().getBoxes(TrackBox.class);
        boolean isError = false;
        for (TrackBox trackBox : trackBoxes) {
            TimeToSampleBox.Entry firstEntry = trackBox.getMediaBox().getMediaInformationBox().getSampleTableBox().getTimeToSampleBox().getEntries().get(0);
            // Detect if first sample is a problem and fix it in isoFile
            // This is a hack. The audio deltas are 1024 for my files, and video deltas about 3000
            // 10000 seems sufficient since for 30 fps the normal delta is about 3000
            if (firstEntry.getDelta() > 10000) {
                isError = true;
                firstEntry.setDelta(3000);
            }
        }

        if (isError) {
            Movie movie = new Movie();
            for (TrackBox trackBox : trackBoxes) {
                movie.addTrack(new Mp4TrackImpl(channel.toString() + "[" + trackBox.getTrackHeaderBox().getTrackId() + "]", trackBox));
            }
            movie.setMatrix(isoFile.getMovieBox().getMovieHeaderBox().getMatrix());
            Container out = new DefaultMp4Builder().build(movie);

            //delete file first!
            FileChannel fc = new RandomAccessFile(mPostProcessingFilePath, "rw").getChannel();
            out.writeContainer(fc);
            fc.close();
            deleteFile(mVideoFilePath);
            mListener.onVideoStop(mPostProcessingFilePath);
            return mPostProcessingFilePath;
        }
        mListener.onVideoStop(mVideoFilePath);
        return mFilePath;
    }catch (IOException e){
        mListener.onVideoError("");
        return mPostProcessingFilePath;
    }
}

In the parser function it checks the delta value, if it's bigger than 10000, it processes the recorded video and returns the result. In case if the delta value is less than 10000, it just returns video without processing. For more details, please refer to the link. Hope it helps you.