Node.js and Requestjs: Pipe a Transform stream to Request.js as a file for upload

Question

I am creating a zip file on memory using node-archiver ( https://github.com/archiverjs/node-archiver ). Using archiver's pipe feature, I pipe everything to a Transform stream. I can pipe this stream to a file if needed, but I would like to let Request ( https://github.com/request ) read this stream, so I don't have to get to the filesystem.

Below, my transform stream is called bridge. I am not doing anything special on the Transform (I believe it could also be a PassThrough stream).

var archive = archiver('zip');
archive.pipe(bridge);
var r = request.post(url, function(err, res, body){ .... }
var form = r.form();
form.append('token', <some token>);
form.append('file', bridge, { "filename" : "package.zip", "contentType" : "application/zip" });
archive.append(<some string>, { "name": <some file name> });
archive.finalize();

This doesn't work (it seems the file part is empty). However, if I pipe my bridge Transform stream to a file write stream -- and after stream finishes -- I create a read stream to that file on the form file operation, it works. Of course, because now I have a fully formed file and request can read it (and I wouldn't need a bridge here, just the regular fs stream)

bridge.pipe(fs.createWriteStream(<myfile>));
var archive = archiver('zip');
archive.pipe(bridge);
bridge.on("finish", function(){
    var r = request.post(url, function(err, res, body){ .... }
    var form = r.form();
    form.append('token', <some token>);
    form.append('file', fs.createReadStream(<myfile>), { "filename" : "package.zip", "contentType" : "application/zip" });

}
archive.append(<some string>, { "name": <some file name> });    
archive.finalize();

I wonder if request requires the read stream to be a file stream -- and what else I could be doing wrong here.

Answer 1

I have a similar issue and discovered that the problem is neither with nodejs or requestjs.

This is a limitation with HTTP. You cannot upload a file without a known content length, which you cannot know from a transformation until after it's done. See this: HTTP POST: content-length header required?

The only way you can work around this by uploading small chunks at a time, with the readable event. Though, this would require your target api to allow chunked or multipart uploads.