What is back pressure on stream processing?

Question

I started learning NodeJS, and streams seems to be something that people use a lot. In most of the documentation that I had read there are mentions of the "back-pressure problem" that occurs when you are processing big sized files but I haven't found a clear explanation of what exactly this problem is. Also I have read that using pipes can help with this problem, but how exactly do pipes fix the back-pressure problem?

Thanks for any explanation in advance.

Answer 1

Backpressure is when you write to a stream faster than the other process can handle/consume Using Pipes, you can control the flow, pause and resume the stream; Here is an example implementing backpressure in nodejs

var http = require('http'),
    fs = require('fs');

var server = http.createServer(function(request, response) {
  var file = fs.createWriteStream('upload.jpg'),
      fileBytes = request.headers['content-length'],
      uploadedBytes = 0;

  request.on('data', function(chunk) {
    uploadedBytes += chunk.length;
    var progress = (uploadedBytes / fileBytes) * 100;
    response.write('progress: ' + parseInt(progress, 10) + '%\n');

    var bufferOK = file.write(chunk);

    if (!bufferOK) {
      request.pause();
    }
  });

  file.on('drain', function() {
    request.resume();
  });

  request.on('end', function() {
    response.end('upload complete\n');
  });

});

server.listen(8080);

Ben foster solution - Souce: https://gist.github.com/benfoster/9543337