Jerky sound from libretro using libao

Question

I'm trying to create a tiny libretro frontend in C++ using SFML for the graphics, libao for the sound and some parts of the Qt framework.

Here is the logic : using SFML's built-in clock system, I refresh at every frame the libretro core, at a given cadency. The core then first polls for the inputs and gives me the video and audio output for the generated frame.

The problem is with the audio : every frame, the core gives me enough audio to match its sample rate (for the SNES it's 32000 samples per second, for example). So, each time I get the audio callback, I simply use ao_play to play everything needed.

Here is the code run at every frame ; I've initialized libao using the parameters given by the core : signed 16bits integers, good sample rate and 2 channels.

size_t RetroCore::processAudioSampleBatch(const int16_t *data, size_t frames)
{
    ao_play(&aoDevice, (char*)data, frames*4);
    return frames; //just to inform the core
}

But the result is quite jerky, like if there was some latency between each frame. The sound is accurate, but it sounds like very low quality and poor outputted. I've been told that I need a stronger sound implementation, what does it mean ? Do I need to resample the sound, or make an external thread ?

Thanks !