Implementing VOIP over udp, what is the approach to take when the player cannot cope up with received packet's speed

Question

I am implementing p2p voice over IP on android, using UDP, openSL and some audio compression. The voice chat works good when 2 peers are connected, and the lag is steadily about 100ms. When a third peer connects, one or more peer progressively accumulate lag, reaching 15+ seconds in some cases.

My guess is that for some reason processing speed can't cope up with the receiving speed (packets gets accumulated in the audio buffer). What is the correct approach to take in this case? Should I discard some packets so that the lag goes back to a small value?

I don't think I have a problem with the application implementation, I have one thread for UDP receiving (no sleep, only I/O) and enqueuing audio for play, one for voice recording and UDP transmission(no sleep, only I/O) and some other threads that do unrelated things. Is this a common situation in VOIP, or is there a problem with this design?

EDIT: I was previously mixing the sound with OpenSL, but now I switched to the simpler android API which seems to alleviate the problem but does not solve it.

This gives the image of the relevant steps I'm taking:

// setup
private AudioTrack track[] = new AudioTrack[MAX_CHANNELS];

for( int i=0; i<MAX_CHANNELS; i++ ) {
        track[i] = new AudioTrack(AudioManager.STREAM_VOICE_CALL, SAMPLING_RATE, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT, outBufferSize, AudioTrack.MODE_STREAM);
        track[i].play();
}

// [...]
// -- RECEIVE THREAD --
DatagramPacket datagram = socket.receiveMessage();
// ... determine channel by ip
track[channel].write(datagram.getData(), datagram.getOffset(), datagram.getLength());

Answer 1

Since AudioTrack.write is blocking, you should make sure you have one thread for every channel you are writing to, so that write calls don't get stuck waiting for other channels to complete.