How do I play a stream of data with custom but partially supported MimeType in browser?

Question

Bringing this over from softwareengineering. Was told this question may be better for stackoverflow.

I am sending a video stream of data to another peer and want to reassemble that data and make that stream the source of a video element. I record the data using the npm package RecordRTC and am getting a Blob of data every 1s.

I send it over an WebRTC Data Channel and initially tried to reassemble the data using the MediaSource API but turns out that MediaSource doesn't support data with a mimetype of video/webm;codecs=vp8,pcm. Are there any thoughts on how to reassemble this stream? Is it possible to modify the MediaSource API?

My only requirement of this stream of data is that the audio be encoded with pcm but if you have any thoughts or questions please let me know!

P.S. I thought opinion based questioned weren't for stackoverflow so that's why I posted there first.

Answer 1

The easiest way to handle this is to proxy the stream through a server where you can return the stream as an HTTP response. Then, you can do something as simple as:

<video src="https://example.com/your-stream"></video>

The downside of course is that now you have to cover the bandwidth cost, since the connection is no longer peer-to-peer.

What would be nice is if you could use a Service Worker and have it return a faked HTTP response from the data you're receiving from the peer. Unfortunately, the browser developers have crippled the Service Worker standards by disabling it if the user reloads the page, or uses privacy modes. (It seems that they assumed Service Workers were only useful for caching.)

Also, a note on WebRTC... what you're doing is fine. You don't want to use the normal WebRTC media streams, as not only are they lossy compressed, but they will drop segments to prioritize staying realtime over quality. This doesn't sound like what you want.

I've been wondering this - is the raw mediastream returned from something like getusermedia what format is that in?

The MediaStream is the raw data, but it isn't accessible directly. If you attach the MediaStream to a Web Audio API graph, whatever format the sound card captured in is converted to 32-bit floating point PCM. At this point, you can use a script processor node to capture the raw PCM data.