How to convert ArrayBuffer sent from websocket into a MediaStream, and then transmit it to webrtc server via peerConnection.addTrack?

Question

The problem:

I have audio binary data, sent from websocket, in ArrayBuffer or Uint8Array format. Let's name it as wsAudioData.

I want to transmit this wsAudioData to my webrtc server, and I can acheive this by utilizing the RTCPeerConnection.addTrack(track: MediaStreamTrack, ...streams: MediaStream[]): RTCRtpSender api.

However, addTrack accepts a track and streams, which are MediaStreamTrack and MediaStream[]. So I have to convert my wsAudioData into MediaStream first.

---

Here's what I have tried:

I tried AudioContext and audioContext.createMediaStreamDestination.

this.audioContext = new AudioContext({ sampleRate: sampleRate });
this.mediaStreamDestination = this.audioContext.createMediaStreamDestination();

When receiving wsAudioData, I will try to play this audio data.

const bufferSource = this.audioContext.createBufferSource();
const audioBuffer = convertUint8ArraysToFloat32Array(wsAudioData);
const buffer = this.audioContext.createBuffer(
        1,
        audioBuffer.length,
        this.audioContext.sampleRate,
      );
buffer.copyToChannel(audioBuffer, 0);
bufferSource.buffer = buffer;

// Connect sourceNode to MediaStreamDestinationNode
bufferSource.connect(this.mediaStreamDestination);
bufferSource.start();

When the websocket is on, I will connect mediaStreamDestination with my peerConnection:

this.peerConnection.addTrack(
  this.mediaStreamDestination.stream.getAudioTracks()[0],
  this.mediaStreamDestination.stream,
);

---

Issues for my solution above:

• The conversion is CPU intensive, when running this code, I can feel my M2 mac is heating.
• When multiple wsAudioData sent from websocket, multiple audio may be played at the same time. (I can partly solve this by using a cache array, when the array's length keeps the same for like 500ms, I play all the audio data in the cache array)

Is there any other way for converting the wsAudioData into a MediaStream seamlessly? Thanks for you reading.

Answer 1

If you don't mind using a Chromium-only API you could use a MediaStreamTrackGenerator to create a MediaStreamTrack without using an AudioContext.

const mediaStreamTrackGenerator = new MediaStreamTrackGenerator('audio');

That mediaStreamTrackGenerator can then be used like any other MediaStreamTrack. You can push data to it by using the WritableStream exposed via its writable property.

It expects the data to be in the AudioData format defined by Web Codecs.