NAudio ASIO Playback glitches

Question

I have an application using AsioOut from the NAudio library to play audio files at low latency.

At certain points in time with very high CPU load glitches appear in the audio. This is to be expected and ok in that application. However, the playback does not always recover after everything goes back to normal. The song keeps playing in regular speed, but heavily distorted with lots of noise (interestingly it is possible to push it back to regular operation with another CPU-load spike).

Has anybody yet experienced something similar?

Unfortunately, I was not able to get any indication from AsioOut that something is wrong. I checked all the outputs provided; however, they are unchanged between regular and distorted operation.

I excluded most of my code as I can also get into the distorted state if I provide an empty sample provider:

class SampleProvider : ISampleProvider
{
   public int Read(float[] buffer, int offset, int count)
   {
      return 0;
   }
}

class ASIOInterface
{
   private void connect(int index)
   {
      asioOut = new AsioOut(index);
      asioOut.Init(sampler);
      asioOut.Play();
   }
}

After inducing the fault I exchange the sample-provider to one with actual music and I can hear the distortions.