Unity Opus library unexpected and weird microphone sounds when receiving from server

Question

I'm working on a proximity voice chat system in a multiplayer game that is using Riptide networking solution, and I found a demo project for using Opus: https://github.com/ludos1978/OpusDotNet I copied the code from the demo and changed it to work for my game and with the Riptide networking systems but I get these weird noises after the data is sent back to client: https://youtu.be/cW-hdyS1g4c

also the noise is not always the same. its really random and sometimes the voice transfers clearly and sometimes its the same as in the video and the voice seems to be playing two times but actually I said hello only once

Client Send Voice code: ` /* AudioClip micSound; public AudioSource player;*/

// device id of the microphone used to record sound
public int micDeviceId = 0;

// the audio source for the mic, used for recording sound
AudioSource audiorecorder;
// buffers to receive and send audio data
List<float> micBuffer;
// size of a network package of opus, audio needs to be split in equal size packages and sent
int packageSize;

// decoder and encoder instances of opus
OpusEncoder encoder;
bool useNoiseReducer;

        // other values then stereo will not yet work
POpusCodec.Enums.Channels opusChannels = POpusCodec.Enums.Channels.Stereo;
// other values then 48000 do not work at the moment, it requires and additional conversion before sending and at receiving
// also osx runs at 44100 i think, this causes also some hickups
POpusCodec.Enums.SamplingRate opusSamplingRate = POpusCodec.Enums.SamplingRate.Sampling48000;

private void Start()
{
    micDeviceId = SettingsManager.settings.microphoneDevice;
    Initialize();
}

void Initialize()
{
    micBuffer = new List<float>();

    //data = new float[samples];
    encoder = new OpusEncoder(opusSamplingRate, opusChannels);
    encoder.EncoderDelay = POpusCodec.Enums.Delay.Delay20ms;

    // the encoder delay has some influence on the amout of data we need to send, but it's not a multiplication of it
    packageSize = encoder.FrameSizePerChannel * (int)opusChannels;
    //dataSendBuffer = new float[packageSize];

    audiorecorder = GetComponent<AudioSource>();
    audiorecorder.loop = true;
    audiorecorder.clip = Microphone.Start(
        Microphone.devices[micDeviceId],
        true,
        1,
        AudioSettings.outputSampleRate);
    audiorecorder.Play();
}

// this handles the microphone data, it sends the data and deletes any further audio output
void OnAudioFilterRead(float[] data, int channels)
{
    // add mic data to buffer
    if (useNoiseReducer)
    {
        Denoiser denoiser = new Denoiser();
    }

    if(!SettingsManager.settings.muteMicrophone)
        micBuffer.AddRange(data);

    // clear array so we dont output any sound
    for (int i = 0; i < data.Length; i++)
    {
        data[i] = 0;
    }
}

public void ChangeMicrophoneDevice(int device)
{
    micDeviceId = device;
    Initialize();
}

private void Update()
{
    if (Input.GetKeyDown(KeyCode.R)) useNoiseReducer = !useNoiseReducer;

    //if (NetworkManager.instance.client != null && NetworkManager.instance.client.IsConnected)
        SendData();
}

void SendData()
{
    // take pieces of buffer and send data
    while (micBuffer.Count > packageSize)
    {
        byte[] encodedData = encoder.Encode(micBuffer.GetRange(0, packageSize).ToArray());

        if (NetworkManager.instance.client != null && NetworkManager.instance.client.IsConnected && !SettingsManager.settings.muteMicrophone)
            DistributeData(encodedData);

        micBuffer.RemoveRange(0, packageSize);
    }
}

void DistributeData(byte[] encodedData)
{
    Message message = Message.Create(MessageSendMode.Unreliable, ClientToServerId.microphone);
    message.AddBytes(encodedData);
    NetworkManager.instance.client.Send(message);
}

`

Client Receive Voice Code: ` public AudioSource audioplayer;

Entity entity;
List<float> receiveBuffer;
// size of a network package of opus, audio needs to be split in equal size packages and sent
int packageSize;

OpusDecoder decoder;

// other values then stereo will not yet work
POpusCodec.Enums.Channels opusChannels = POpusCodec.Enums.Channels.Stereo;
// other values then 48000 do not work at the moment, it requires and additional conversion before sending and at receiving
// also osx runs at 44100 i think, this causes also some hickups
POpusCodec.Enums.SamplingRate opusSamplingRate = POpusCodec.Enums.SamplingRate.Sampling48000;

private void Start()
{
    Initialize();
}

void Initialize()
{
    entity = GetComponent<Entity>();
    // playback stuff
    decoder = new OpusDecoder(opusSamplingRate, opusChannels);

    receiveBuffer = new List<float>();

    // setup a playback audio clip, length is set to 1 sec (should not be used anyways)
    AudioClip myClip = AudioClip.Create("MyPlayback", (int)opusSamplingRate, (int)opusChannels, (int)opusSamplingRate, true, OnAudioRead, OnAudioSetPosition);
    audioplayer.loop = true;
    audioplayer.clip = myClip;
    audioplayer.Play();
}

public void ReceiveData(byte[] encodedData)
{
    float[] decodedData = decoder.DecodePacketFloat(encodedData);

    for (int i = 0; i < decodedData.Length; i++)
    {
        decodedData[i] *= entity.isMute ? 0f : entity.voiceVolume;
    }

    // the data would need to be sent over the network, we just decode it now to test the result
    receiveBuffer.AddRange(decodedData);
}

// this is used by the second audio source, to read data from playData and play it back
// OnAudioRead requires the AudioSource to be on the same GameObject as this script
void OnAudioRead(float[] data)
{
    int pullSize = Mathf.Min(data.Length, receiveBuffer.Count);
    float[] dataBuf = receiveBuffer.GetRange(0, pullSize).ToArray();
    dataBuf.CopyTo(data, 0);
    receiveBuffer.RemoveRange(0, pullSize);
}

`

I tried playing around with the OnAudioRead function but nothing changed

also I tried making a package system that will wait till the right amount of data is received and the noise stayed the same.