How to deal with `CHUNK_DEMUXER_ERROR_APPEND_FAILED` MediaSource error on Chrome?

Question

I'm trying to write a single-page application in HTML that streams YouTube videos. As some may know, YouTube keeps high-quality video/audio as separate streams. I've already figured out how to get video/audio URLs to stream from. I need to make a <video> element play the combined streams. This is what I've achieved so far with the MediaSource API:

/**
 * @param {string} url 
 * @param {SourceBuffer} sourceBuffer 
 */
const streamToSourceBuffer = (url, sourceBuffer) =>
    fetch(url)
        .then(({ body }) =>
            body.pipeTo(new WritableStream({
                write(chunk) {
                    if (!sourceBuffer.updating)
                        return sourceBuffer.appendBuffer(chunk);

                    const onupdate = () => {
                        if (sourceBuffer.updating) return;
                        sourceBuffer.appendBuffer(chunk);
                        sourceBuffer.removeEventListener('update', onupdate);
                    };

                    sourceBuffer.addEventListener('update', onupdate);
                }
            }))
        );

const mse = async () => {
    const { formats } = await getInfo('05MQCSWV4fU');

    const audioFormat = highestBitrate(audioOnly(formats));
    const videoFormat = highestBitrate(videoOnly(formats));

    const mediaSource = new MediaSource();
    mediaSource.onsourceopen = () => {
        if (mediaSource.sourceBuffers.length >= 2)
            return;
        streamToSourceBuffer(audioFormat.url, mediaSource.addSourceBuffer(audioFormat.mimeType));
        streamToSourceBuffer(videoFormat.url, mediaSource.addSourceBuffer(videoFormat.mimeType));
    };

    player.src = URL.createObjectURL(mediaSource);
    player.onerror = () => console.error(player.error);
};

I watched the Network tab of Chrome DevTools, and made some observations. Initially the audio/video downloads are at the same speed. Once the audio download finishes, the browser uses the remaining bandwidth to speedup the video download. Only when around 7-8MB of the video is downloaded, I get a MediaError from my <video> element:

MediaError {
    code: 3,
    message: 'CHUNK_DEMUXER_ERROR_APPEND_FAILED: Failed to prepare video sample for decode'
}

With video ID OLIdxa4VNiU, the MediaError has a different message:

CHUNK_DEMUXER_ERROR_APPEND_FAILED: RunSegmentParserLoop: stream parsing failed. append_window_start=0 append_window_end=inf

I did even further digging and found these messages in the Media tab of DevTools: My first thought was "maybe the video stream is just corrupted", but once I streamed a lower bitrate version of the video, no errors occurred:

const audioFormat = highestBitrate(audioOnly(formats));
const videoFormat = lowestBitrate(videoOnly(formats));

To add on to these problems, sometimes the video/audio downloads are throttled by each other, as in they take turns receiving data every half a second. Normally they are concurrent downloads. This is sometimes fixable by refreshing my HTML document.

Anyone experienced in web/media please leave some advice.