Unable to use one readable stream to write to two different targets in Node JS

Question

I have a client side app where users can upload an image. I receive this image in my Node JS app as readable data and then manipulate it before saving like this:

uploadPhoto: async (server, request) => {  

        try {         

            const randomString = `${uuidv4()}.jpg`;
            
            const stream = Fse.createWriteStream(`${rootUploadPath}/${userId}/${randomString}`);

            const resizer = Sharp()
                                .resize({
                                    width: 450
                                });

            await data.file
                    .pipe(resizer)
                    .pipe(stream);

This works fine, and writes the file to the projects local directory. The problem comes when I try to use the same readable data again in the same async function. Please note, all of this code is in a try block.

        const stream2 = Fse.createWriteStream(`${rootUploadPath}/${userId}/thumb_${randomString}`);

        const resizer2 = Sharp()
                            .resize({
                                width: 45
                            });

        await data.file
                .pipe(resizer2)
                .pipe(stream2);

The second file is written, but when I check the file, it seems corrupted or didn't successfully write the data. The first image is always fine.

I've tried a few things, and found one method that seems to work but I don't understand why. I add this code just before the I create the second write stream:

    data.file.on('end', () => {
        console.log('There will be no more data.');
    });

Putting the code for the second write stream inside the on-end callback block doesn't make a difference, however, if I leave the code outside of the block, between the first write stream code and the second write stream code, then it works, and both files are successfully written.

It doesn't feel right leaving the code the way it is. Is there a better way I can write the second thumb nail image? I've tried to use the Sharp module to read the file after the first write stream writes the data, and then create a smaller version of it, but it doesn't work. The file doesn't ever seem to be ready to use.

Answer 1

You have 2 alternatives, which depends on how your software is designed.

If possible, I would avoid to execute two transform operations on the same stream in the same "context", eg: an API endpoint. I would rather separate those two different tranform so they do not work on the same input stream.

If that is not possible or would require too many changes, the solution is to fork the input stream and the pipe it into two different Writable. I normally use Highland.js fork for these tasks.

Please also see my comments on how to properly handle streams with async/await to check when the write operation is finished.