Converting MIDI file to raw audio using a software synth

Question

I'm trying to dynamically generate a small MP4 audio+video file directly from my Android app.

My original plan of attack:

1. The user enters some basic song data (a chord progression, etc) and the app builds a MIDI file.

2. The system builds chord diagrams for each chord, and using a MIDI reader it generates the animations frames array that is timed to the MIDI

3. Convert the MIDI into a raw PCM audio data <-- this S.O. question is specific to this point

4. Apply the raw audio to the animation frames - and encode the audio and video frames into an MP4

5. Provide the resulting MP4 video to the user with standard playback controls.

The App already builds the MIDI file based on user inputs including tempo, instrument, notes, etc. This part is done easily and the MIDI file is being generated correctly. This MIDI plays fine from Android's MediaPlayer. (step 1)

The animation frames are also being created correctly by reading back the MIDI file and cross referencing a list of static bitmaps for each chord in the sequence. These frames will become the video... but will be without audio at first. (step 2)

As you can see, the problems with Android MIDI latency is not so much a problem for me because I'm not creating a real-time synthesizer App... I just want to convert the MIDI to some audio format that can then be mixed into a video that is already timed to original the MIDI. (step 3)

---

The problem I'm running into is Step 3.

I understand that I need to use a software MIDI synthesizer to obtain the actual audio output that would be produced from a sequence of MIDI events. However, getting this to work correctly has become a major hurdle. I do not care about the precise quality of the resulting audio, just that it matches closely to what someone would expect if they where using General MIDI samples from a generic device (ala Gravis soundfonts or bulit-in Sovinox sounds, etc).

So...

1. Android MIDI Lib

2. Android Midi Driver using Sonivox EAS Library

My first attempt was with the above two projects... mash them up in such a way that a MIDI file could be converted into a raw PCM data buffer.... it hasn't gone so well yet.

The midi library (1) reads a MIDI file with a real-time listener and sends the events to the Midi Driver (2) which plays the raw PCM data generated by the onboard synthesizer.

(NOTE: A few tweaks were made to the driver so that the entire buffer was stored and only returned once the MIDI reader was finished. This also means that the entire process takes an amount of time equal to length of the song just to convert it because we are "listening" in real-time.)

I have yet to get this to work as I hoped. I wanted to keep this as simple as possible and prefer to use open source projects if at all possible. It would be better if it could do this without relying on real-time listeners.

---

Some other libraries and tools that I have been considering (but may be overkill):

1. Pure Data Library for Android

2. MidiSheetMusic App (with source)

3. Synthesis ToolKit in C++ (STK)

4. Music Synth for Android

5. Crimson SoftSynth

EVEN MORE CONTENDERS (not much research done yet):

jFugue/timidity/audacity/fluidSynth/cSound/jMusic/JSyn/Gervill/Softonic/Caustic/LibGDX/JetPlayer/OpenSL-ES

---

My questions are:

• Am I working down the correct path with the above projects? I'm relatively new to MIDI->PCM conversion so I want to make sure I'm not completely missing something.

• If not, how should I go about converting a MIDI file into some sort of audio format that can then be used in the creation of an MP4 (or any video playback format)?

• Are there other open source projects that may be helpful in this task of converting MIDI 2 raw audio waveforms using Android?

• And are there any examples of such a task already written for use on Android? (i.e. already ported for use with Android JNI, etc)

My hope is that I completely missed something that will make this a trivial task... my assumption is that this is going to take some serious hacking and JNI kung-fu.

I am open to taking the hard road if that's what it takes. Any and all advice would be appreciated.

Answer 1

I'm not an Android user, but I tried many ways to generate WAV files from MIDI, and I'm the more happy about this task, with FluidSynth, which seems to be available as a library for Android, and is the point of this other StackOverflow question: Using Fluidsynth to play notes from SoundFonts on Android.

You just need to know about sound‑fonts (searching the web will easily tell you) and choose a good one which is not too big (otherwise, you may have lack of memory issue).

Also check the sound‑font covers all the instruments you want to be playable.

Answer 2

If you're running in a JS or Node environment, you should be able to use mudcube's MIDI.js (or my ES6 port as midicube) to use the Player object to convert the MIDI to webaudio output and easily play that. Or if you want to save it, you could capture the stream and convert it to any number of formats (probably including mp3 but I've never tried that). If time is not of the essence in the conversion, the second part should be easy, but it will take the running time of the MIDI to convert. It should be possible to speed up the playback speed and the playback of the samples to make the conversion happen in 2 or 4 times speed and then slow down the file later, but I have not tried that.

If you have lower-level access to libraries, @Hibou57's suggestion of using FluidSynth is best.

Either way you'll need some soundfonts. Fluid-GM works well for lower quality but acceptable sound.