How to convert a Buffer to a video with fluent-ffmpeg and write the output to a WritableStream

271 views Asked by At

For some context, I need to quickly render a small video (6.5 seconds long, at 15 FPS), and send it in a Discord channel with discord.js. I don't want to ever have to write anything to disk, because that'll slow it down and I just don't need it downloaded. So far, I was able to write the video to disk, but now I want to skip that step and send the video Buffer straight to discord.js. I was also able to output the video from ffmpeg to Discord as an audio file, but when I try to use the .mp4 format, I get a "Conversion failed!" error.

I rendered the individual frames for the video using the canvas module, and export all of them as pngs using canvas.toBuffer("image/png");, and then push all the frames to an array. Then I combine the frames to one Buffer using Buffer.concat(), and then create a ReadableStream from the nodejs stream module. I also needed to write a custom WritableStream class that implements the _write method. Here's the class:

class MyWritable extends Stream.Writable {
    constructor() {
        super();
        this.buffer = Buffer.from([]);
    }

    _write(chunk, encoding, callback) {
        this.buffer = Buffer.concat([this.buffer, chunk]);
        callback();
    }
}

And here's how I implement everything using fluent-ffmpeg:

const allFrames = Buffer.concat(framesArray);
const readable = Stream.Readable.from(allFrames);
const writable = new MyWritable();

const output = await (new Promise((resolve, reject) => {
    ffmpeg()
        .input(readableStream)
        .inputOptions([
            `-framerate 15`,
        ])
        .input("path_to_audio_file.mp3")
        .videoCodec("libx264")
        .format("mp4")
        .outputOptions([
            "-pix_fmt yuv420p"
        ])
        .duration(6.5)
        .fps(15)
        .writeToStream(writable)
        .on("finish", () => {
            // this is never reached, but it should resolve the promise with all the data that was written to the WritableStream (which should be the video)
            resolve(writable.buffer);
        })
        .on("error", (err) => {
            // this is never reached also
            reject(err);
        })
}))

// then I try to send the output buffer to Discord as an attachment, but I don't ever get here anyway

And here's the error I get:

node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: ffmpeg exited with code 1: Conversion failed!

    at ChildProcess.<anonymous> (/Users/isaac/Documents/Github/DiscordBot/node_modules/fluent-ffmpeg/lib/processor.js:182:22)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)
Emitted 'error' event on FfmpegCommand instance at:
    at emitEnd (/Users/isaac/Documents/Github/DiscordBot/node_modules/fluent-ffmpeg/lib/processor.js:424:16)
    at endCB (/Users/isaac/Documents/Github/DiscordBot/node_modules/fluent-ffmpeg/lib/processor.js:544:13)
    at handleExit (/Users/isaac/Documents/Github/DiscordBot/node_modules/fluent-ffmpeg/lib/processor.js:170:11)
    at ChildProcess.<anonymous> (/Users/isaac/Documents/Github/DiscordBot/node_modules/fluent-ffmpeg/lib/processor.js:182:11)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)

Like I said earlier, if I change the format to an audio format, it exports perfectly fine. But I have no idea where the problem is with converting to a video. Thanks in advance.

0

There are 0 answers