Android, adding Audio track to Muxer

Question

I am trying to add Audio to video created by below open source project

Specifically to https://github.com/madisp/trails/blob/master/app/src/main/java/com/madisp/trails/CaptureService.java

I need to get audio from MIC and write it to encoding file as audio track. At the moment file encoded with Muxer has only video track.

I can get audio from MIC without any issues below

int nChannels = 1;
int minBufferSize = AudioRecord.getMinBufferSize(44100, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT) * 2;
AudioRecord aRecorder = new AudioRecord(MediaRecorder.AudioSource.MIC, 44100, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, minBufferSize);

short[] buffer = new short[44100 * nChannels];
aRecorder.startRecording();
int readSize = 0;

while (recording) {
    readSize = aRecorder.read(buffer, 0, minBufferSize);
    if (readSize < 0) {
        break;
    } else if (readSize > 0) {
        // do stuff with buffer
    }
}
aRecorder.stop();
aRecorder.release();

but i am not sure how to incorporate it in to (https://github.com/madisp/trails/blob/master/app/src/main/java/com/madisp/trails/CaptureService.java)

while (running) {
    int index = avc.dequeueOutputBuffer(info, 10000);
    if (index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
        if (track != -1) {
            throw new RuntimeException("format changed twice");
        }
        track = muxer.addTrack(avc.getOutputFormat());
        muxer.start();
    } else if (index >= 0) {
        if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
            // ignore codec config
            info.size = 0;
        }
        if (track != -1) {
            ByteBuffer out = avc.getOutputBuffer(index);
            out.position(info.offset);
            out.limit(info.offset + info.size);
            muxer.writeSampleData(track, out, info);
            avc.releaseOutputBuffer(index, false);
        }
    }
}

And yes, understand I am literally asking for you to write the code but I have not mush expertise on this

Any help appreciated

Thanks

Answer 1

First, use byte[] instead of short[] for the buffer used with AudioRecord - that will simplify things a little.

Then, to encode the received buffer, something like this should work (untested):

while (recording) {
    readSize = aRecorder.read(buffer, 0, minBufferSize);
    if (readSize < 0) {
        break;
    } else if (readSize > 0) {
        boolean done = false;
        while (!done) {
            int index = avc.dequeueInputBuffer(10000);
            if (index >= 0) { // In case we didn't get any input buffer, it may be blocked by all output buffers being full, thus try to drain them below if we didn't get any
                ByteBuffer in = avc.getIndexBuffer(index);
                in.clear();
                in.put(buffer, 0, readSize);
                avc.queueInputBuffer(index, 0, readSize, System.nanoTime()/1000, 0);
                done = true; // Done passing the input to the codec, but still check for available output below
            }
            index = avc.dequeueOutputBuffer(info, 10000);
            if (index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
                if (track != -1) {
                    throw new RuntimeException("format changed twice");
                }
                track = muxer.addTrack(avc.getOutputFormat());
                muxer.start();
            } else if (index >= 0) {
                if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
                    // ignore codec config
                    info.size = 0;
                }
                if (track != -1 && info.size > 0) {
                    ByteBuffer out = avc.getOutputBuffer(index);
                    out.position(info.offset);
                    out.limit(info.offset + info.size);
                    muxer.writeSampleData(track, out, info);
                    avc.releaseOutputBuffer(index, false);
                }
            }
        }
    }
}

I think the normal SW AAC encoder should be ok with you passing an arbitrary number of bytes of audio to it, but in case the encoder is picky, you'd need to pass it recorded data in blocks of 1024 samples (2048 bytes for mono, 4096 bytes for stereo).