Why do I need a callback in my .write() method for the stream API?

Question

This question regards the callback in the Writable.write method of the node.js stream API.

I am in the process of learning about node.js streams, using stream-adventure, which is essentially an interactive tutorial for streams. One of the problem statements is to pipe process.stdin to a custom writable stream that calls console.log('writing' + chunk). My first attempt at a solution was:

const {Writable} = require('stream');

const writer = new Writable({
  write(chunk){
    console.log('writing: ' + chunk);
  }
})

process.stdin.pipe(writer);

And the output (this behavior is internal to the testing that stream-adventure performs) was:

         Actual                                  Expected
   "writing: Screamer"                 ==    "writing: Screamer"
   ""                                  ==    ""
   ""                                  !=    "writing: Weeping Angel"
                                       !=    "writing: Sontaran"
                                       !=    "writing: Racnoss"
                                       !=    "writing: Logopolitan"
                                       !=    "writing: Shansheeth"
                                       !=    "writing: Arcturan"
                                       !=    "writing: Cyberman"
                                       !=    "writing: Terileptil"
                                       !=    "writing: Ancient Lights"
                                       !=    ""
                                       !=    ""
                                       !=    ""
                                       !=    ""
                                       !=    ""
                                       !=    ""
                                       !=    ""
                                       !=    ""
                                       !=    ""

I realize that using stream-adventure confuses the issue, so please bear with me. The stream does not properly reset for the next write. After looking at examples online, I solved the problem using:

const { stdin } = require('process');
const {Writable} = require('stream');

const writer = new Writable({
  write(chunk, encoding, next){
    console.log('writing: ' + chunk);
    next(); 
  }
})

process.stdin.pipe(writer);

The only real difference here is the execution of the callback at the end of the write. However, I do not know why this worked. Looking in at the Writable class API, both the encoding and callback are optional. Presumably the Readable.pipe method is passing some callback to <next>, but this I cannot find it in the docs. What is node.js doing under the hood that causes the streams to stop in the first case, and why does executing the callback in .write() fix the problem?

Answer 1

It looks like I was confused by the difference between the internal _write() method and the public write() method. While it is true that, in general, the write() method does not require a callback, _write() does. _write() uses the callback to indicate that the stream has finished processing the current chunk.

The Node API suggests that when creating a new instance of a Writable stream, you supply the _write(), _writev(), _final() methods to the object. However, when I instantiated the stream, I provided the write() method instead. Interestingly, the API documentation references this form of stream construction as a "simplified" construction.

const { Writable } = require('stream');

const myWritable = new Writable({
  write(chunk, encoding, callback) {
    // ...
  }
});

Note that the above now no longer lists indicates that the callback is optional. I suspect (but have not confirmed) that this syntax automatically sets _write() to match write(). Ultimately this means that I was not calling the callback to signal the end of the chunk processing, and so the writable stream never tried to process the next chunk.