MongoError when uploading a file using mongoose, gridfs-stream and multer

Question

I am running express 4 using multer, gridfs-stream and mongoose with mongodb and I am attempting to upload a file and stream it to gridfs.

The express route that does this is defined as:

app.post('/uploadfile', function (req, res) {
    console.dir(req.files);

    // The mongodb instance created when the mongoose.connection is opened
    var db = mongoose.connection.db;

    // The native mongo driver which is used by mongoose
    var mongoDriver = mongoose.mongo;

    // Create a gridfs-stream
    var gfs = new Gridfs(db, mongoDriver);

    var file = req.files.myFile;

    var fileId = new ObjectId();

    console.log("Creating WriteStream");
    var writeStream = gfs.createWriteStream({
        _id: fileId,
        filename: file.originalname,
        mode: 'w',
        content_type: file.mimetype,
        metadata: {
            id: '123',
            number: '2',
            name: "Kenny Erasmuson"
        }
     });
     console.log("Created WriteStream");
     req.pipe(writeStream);
     console.log("Finished!");
});

Once the express app is running, a file is selected and uploaded (via an HTML multipart/form-data form), and the output from the node server is:

$ node server.js 
Listening on port 8001
{ myFile: 
    { fieldname: 'myFile',
    originalname: 'kenny-credit-rating.pdf',
    name: '1082e5071ede1002c4ae5be6123226d8.pdf',
    encoding: '7bit',
    mimetype: 'application/pdf',
    path: 'uploads/1082e5071ede1002c4ae5be6123226d8.pdf',
    extension: 'pdf',
    size: 110782,
    truncated: false,
    buffer: null } 
}
Creating WriteStream
Created WriteStream
Finished!
/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/co nnection/base.js:246
    throw message;      
          ^
MongoError: The dollar ($) prefixed field '$conditionalHandlers' in '_id.$conditionalHandlers' is not valid for storage.
at Object.toError (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/utils.js:114:11)
at /Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/collection/core.js:569:27
at /Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1157:7
at /Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1890:9
at Server.Base._callHandler (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/base.js:448:41)
at /Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:481:18
at MongoReply.parseBody (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
at null.<anonymous> (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:439:20)
at emit (events.js:95:17)
at null.<anonymous> (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:201:13)

Has anyone any ideas what is causing the error and how to fix?

Answer 1

In my code:

var fileId = new ObjectId();

console.log("Creating WriteStream");
var writeStream = gfs.createWriteStream({
    _id: fileId,
    filename: file.originalname,
    mode: 'w',
    content_type: file.mimetype,
    metadata: {
        id: '123',
        number: '2',
        name: "Kenny Erasmuson"
    }
 });

I am assigning an ObjectId rather than the string representation of an ObjectId to the _id property of the object passed into gfs.createWriteStream. It turns out this is causing the MongoError in my code.

The fix, discovered here, is to change the line of code causing the issue to be:

_id: fileId.str,

Having done that I then came to the issue of the file not being put into the fs.chunks mongodb collection although the metadata does end up in the fs.files mongodb collection. AddieD's answer here starts to address this :)

Answer 2

The problem lies in the fact that you aren't piping just a file request to gridfs-stream. Using Multer as a middleware with catch any multipart/form-data post request. As the stream comes in, Multer (built on Busboy) watches for the on('field') and on('file') events and parses accordingly. What you are piping to Multer is not just a file.

This bit of code works fine because Multer has indeed parsed out for you req.files and req.body by this point:

   // Create a gridfs-stream
   var gfs = new Gridfs(db, mongoDriver);

   var file = req.files.myFile;

   var fileId = new ObjectId();

   console.log("Creating WriteStream");
   var writeStream = gfs.createWriteStream({
        _id: fileId,
        filename: file.originalname,
        mode: 'w',
        content_type: file.mimetype,
        metadata: {
            id: '123',
            number: '2',
            name: "Kenny Erasmuson"
        }
     });
     console.log("Created WriteStream");

But where your code runs into problems is here for reasons mentioned above:

 req.pipe(writeStream);

I have yet to find a way to stream a multipart/form-data post request with more than just file uploads straight to GridFS. If your post request doesn't contain anything other than the file (meaning no other html input fields in the form), then you might want to consider removing Multer from the middleware at least for this route.

For my use case, I require the ability to receive an html form post with text inputs along with the file upload (I store metadata about the file on upload). Here is how I accomplished what I needed with Multer:

var uploadImg = function(req,res) {
      var writestream = gfs.createWriteStream({
        filename: req.files.file.name,
        mode:'w',
        content_type:req.files.file.mimetype,
        metadata:req.body,
      });
    fs.createReadStream(req.files.file.path).pipe(writestream);

    writestream.on('close', function (file) {
        res.send("Success!");
        fs.unlink(req.files.file.path, function (err) {
          if (err) console.error("Error: " + err);
          console.log('successfully deleted : '+ req.files.file.path );
        });
    });

};

By default, Multer will store your files on disk. One quick solution is to just create a readstream and stream it right back into GridFS. Once the writing is completed, remove the tmp file.

As a side note, there seems to be some thought going around that it's best to use Multer as middleware only on the routes that need it. You can read more on that thought on the last section of this.

Update

I think I am close to finding a way to stream directly to GridFS with metadata by use of Skipper. I'm in the process of seeing if I can't get some updates going to skipper-gridfs that will take the text inputs on the html form and set those as metadata, etc. The tweak to the skipper-gridfs seems pretty minor. I'll update when that gets flushed out. If you look at skipper be sure you recognize the order of your html form inputs (if you have any) does matter.