What is the best way to convert files with a heroku deployed app?

Question

I am writing a Discord bot and it's deployed to heroku. I have a function of the bot that will take in a video file and convert it from whatever format it's in to .mp4, then the bot replies with the file embedded.

Locally, everything runs OK, and I know that Heroku uses an ephemeral file system, but considering I don't really care about persisting these files, I figured it would be fine to create them on disk, reply, and then delete them.

However, whenever I run the command on the deployed bot I am receiving this error Error: ENOENT: no such file or directory, stat '/app/dest/videostoconvert/851730039589306388.mp4'

const getAndConvertVideoFile = async (
    message: Discord.Message,
    url: string,
    inputPath: string,
    outputPath: string
) => {
    try {
        if (!fs.existsSync(path.join(__dirname, "../../", "videostoconvert"))) {
            const mkDirPromise = util.promisify(fs.mkdir);
            await mkDirPromise(path.join(__dirname, "../../", "videostoconvert"));
        }
    } catch (error) {
        console.log("ERROR CREATING DIRECTORY");
        console.error(error);
    }

    const response = await axios({
        method: "get",
        url: url,
        responseType: "stream"
    });

    console.log("INPUT PATH", inputPath);
    // INPUT PATH /app/dest/videostoconvert/851730039589306388.m4v
    console.log("OUTPUT PATH", outputPath);
    // OUTPUT PATH /app/dest/videostoconvert/851730039589306388.mp4
    try {
        const diskWriteStream = fs.createWriteStream(inputPath);
        response.data.pipe(diskWriteStream);

        diskWriteStream.on("error", (error) => {
            console.log("DISK WRITE ERROR");
            console.error(error);
        });

        diskWriteStream.on("finish", async function () {
            hbjs.spawn({ input: inputPath, output: outputPath })
                .on("error", (error: any) => {
                    console.error("THERE WAS AN ERROR", error);
                    console.log("There was an error", error);
                })
                .on("progress", (progress: any) => {
                    console.log("Percent complete: %s, ETA: %s", progress.percentComplete, progress.eta);
                })
                .on("complete", async () => {
                    console.log("COMPLETE");
                    // seems to be where it fails, heroku logs "COMPLETE" and then the error
                    const attachment = new Discord.MessageAttachment(outputPath);
                    

                    await message.reply(attachment);
                });
        });
    } catch (error) {
        console.log("ERROR GETTING AND CONVERTING FILE");
        console.error(error);
    }
};

I've added some comments in the code that may help. It seems to me like that means that the directory doesn't exist, but I'm creating the directory if it doesn't exist earlier in the code so I'm not sure what is causing the issue. Also, the failure point seems to happen at the hbjs.spawn.on("complete", ...) line based on the Heroku console output, so the directory seems to be created for the purposes of writing the file after it's downloaded.

I will say, I'm not opposed to using S3 to host the files instead of writing directly to the heroku disk, but I'm not sure how I'd convert them in that case, sine hbjs seems to expect an on-disk path for writing out the converted file.

Answer 1

My suspicion is that this is related to the fact internally, handbrake-js spawns a separate process (the HandbrakeCLI binary) to produce the output (an encoded video file). The Node.js process then needs to stay alive for enough ticks to allow the child process to complete.

Maybe Heroku is terminating the Node.js process before it gives fs and/or handbrake-js time to finish.

I personally would try switching from using the event-based interfaces of fs and handbrake-js (i.e. listening for events) to using async methods and Promise-yielding interfaces. See fs/promises and the .run() method of handbrake-js.