how to use node transform to transform one object into another

Question

I'm new to Node streams and trying to figure out why things aren't working as expected. I'm just practicing and want to do a silly transform.

interface MyTransformStreamOptions {
  [key: string]: any
}

class MyTransformStream extends Transform {
  constructor(options) {
    options.readableObjectMode = true;
    options.writableObjectMode = true;

    super(options);
  }

  _transform(chunk, encoding, callback) {
    console.log('encoding: ' + encoding);

    // chunk should be an object here, due to
    // readableObjectMode = true right?

    const newObj = {};

    for (const [key, value] of Object.entries(chunk)) {
      console.log(`[key, value]: [${key}, ${value}]`);
      newObj['aa' + key] = 'bb' + value;
    }

    callback(null, newObj);
  }
}

const myTransformStream = new MyTransformStream({} as MyTransformStreamOptions);

process.stdin
  .pipe(myTransformStream)
  .pipe(process.stdout);

Finally I run the script and input this via stdin:

{ car: 'honda', truck: 'mazda' }

and also I've tried:

"{ car: 'honda', truck: 'mazda' }"

But the loop treats the chunk as a string and not an object. So I see output like:

[key, value]: [0, 123]
[key, value]: [1, 32]
...etc...

I've tried various combinations of:

options.readableObjectMode = true/false;
options.writableObjectMode = true/false;

And also in _transform() using:

const obj = JSON.parse(chunk.toString());

but obj is never an object and is always a string.

My goal is imply to receive an object into my transform stream, alter it, and output it. But I get stuck dealing with strings only that I cannot convert to objects even with JSON.parse().

Where am I going wrong here?

Update - Partial Solution

The problem was with my use of JSON.parse(). I was trying to parse:

{ car: 'honda', truck: 'mazda' }

Which is valid JS, but not valid JSON, duh!

However, the bigger issue still remains. I've set the Transform stream to: writableObjectMode = true, but when I do:

callback(null, newObj);

I get the error:

TypeError: Invalid data, chunk must be a string or buffer, not object

If I do:

callback(null, JSON.stringify(newObj));

things works as expected.

However, shouldn't this work without stringifying given that I've set things up in object mode?

Answer 1

Object mode does not transform the stream from a buffer (it's a buffer not a string) to an object, it just lets the type of data be types other than buffer or string. You need to use a transform stream to convert the stdin buffer into an object such as:

const ndjson = require('ndjson');

. . .

process.stdin
  .pipe(ndjson.parse())
  .pipe(myTransformStream)
  .pipe(process.stdout);

Answer 2

So the main problems with my code are the use of stdin and stdout. They are streams that operate in string/buffer mode. Using them as is, piping stdin to a stream will be piped in string/buffer mode not object mode. The same applies to stdout.

I tested my hypothesis by creating a readable stream in object mode that emulated my stdin input I was inputting via the _read() method and then having my transform stream transform the chunk as an object that the readable stream piped into it.

The pipe to standard out was still failing as it accept objects out of the box. I confirmed this by creating another transform stream that simply transformed the object to a string via JSON.stringify() and then stdout was happy again.

All the problems I was having with 'read/write object modes' was simply because I had stdin and stdout on both sides of the my transform stream.