Making use of MediaCodec API 21 to record video and audio

1.6k views Asked by At

I've been messing around with MediaCodec and MediaMuxer for a while now to record from a virtual display. I got this working nicely. Now I want to record sound from the microphone and add that to the video. I have found dosens of examples that does exactly what I want, however all these examples are making use of the deprecated parts of the MediaCodec. I am really confused how to refactor to the new api.

To start of I got a thread reading bytes from an AudioRecord like this.

while (running && recording) {
            audioPresentationTimeNs = System.nanoTime();

            iReadResult = mAudioRecord.read(mTempBuffer, 0, SAMPLES_PER_FRAME);
            if(iReadResult == AudioRecord.ERROR_BAD_VALUE || iReadResult == AudioRecord.ERROR_INVALID_OPERATION)
                Log.e(TAG, "audio buffer read error");

            // send current frame data to encoder
            try {

This wont work as dequeueInputBuffer cant be called async... So what will i need to do to get the right inputBufferIndex here?

                int inputBufferIndex = audioEncoder.dequeueInputBuffer(-1);

                if (inputBufferIndex >= 0) {
                    inputBuffer = audioEncoder.getInputBuffer(inputBufferIndex);
                    inputBuffer.clear();
                    inputBuffer.put(mTempBuffer);

                    if(VERBOSE)Log.d(TAG, "sending frame to audio encoder");
                    audioEncoder.queueInputBuffer(inputBufferIndex, 0, mTempBuffer.length, audioPresentationTimeNs / 1000, 0);
                }
            } catch (Throwable t) {
                Log.e(TAG, "sendFrameToAudioEncoder exception");
                t.printStackTrace();
            }
        }

Then I have my callback which is used by my AudioEncoder that should receive data from AudioRecord:

private class audioEncoderCallback extends MediaCodec.Callback{

Here should I probably receive the inputData and send to onOutputBufferAvailable somehow?

    @Override
    public void onInputBufferAvailable(MediaCodec codec, int index) {
    }

This piece are supposed to deliver the data to the muxer

    @Override
    public void onOutputBufferAvailable(MediaCodec codec, int index, MediaCodec.BufferInfo info) {

        if(muxerStarted) {
            ByteBuffer encodedData = codec.getOutputBuffer(index);

            encodedData.position(info.offset);
            encodedData.limit(info.offset + info.size);

            mMuxer.writeSampleData(audioTrackIndex, encodedData, info);
            if (VERBOSE) Log.d(TAG, "AudioEncoder sent " + info.size + " bytes to mMuxer");

            codec.releaseOutputBuffer(index, false);
        }
    }

    @Override
    public void onError(MediaCodec codec, MediaCodec.CodecException e) {
        Log.d(TAG, "Error: " + e);
    }

Am I thinking right about having both an audioTrackIndex and videoTrackIndex to deliver data into the muxer?

    @Override
    public void onOutputFormatChanged(MediaCodec codec, MediaFormat format) {
        if (VERBOSE) Log.d(TAG, "encoder output format changed: " + format);

        audioTrackIndex = mMuxer.addTrack(format);
        if(videoTrackIndex != -1) {
            muxerStarted = true;
            mMuxer.start();
        }
    }
}
0

There are 0 answers