How to close a https stream in Node.js

Question

I'm loading a .ndjson file via https. I want to close it after reading 100 lines of the file.

  const amount = 100;
  https.get(url, (res) => {
    var { statusCode } = res;

    if (statusCode !== 200) {
      throw new Error(`Request Failed.\n Status Code: ${statusCode}`);
    }

    res.setEncoding('utf8');
    let rows = [];
    res
      .pipe(ndjson.parse())
      .on('data', function (obj) {
        rows.push(obj);
        if (rows.length === amount) {
          this.end();
        }
      })
      .on('end', () => {
        resolve(rows);
      });
  }).on('error', (e) => {
    throw new Error(e.message);
  });

But every way I have tried the close the stream, the same error message appears:

Error: Could not parse row {"username"...
    at DestroyableTransform.parseRow [as mapper] (C:\Users\thoma\Documents\GitHub\test\node_modules\ndjson\index.js:19:28)
    at DestroyableTransform.flush [as _flush] (C:\Users\thoma\Documents\GitHub\test\node_modules\split2\index.js:44:21)
    at DestroyableTransform.<anonymous> (C:\Users\thoma\Documents\GitHub\test\node_modules\readable-stream\lib\_stream_transform.js:138:49)
    at Object.onceWrapper (events.js:314:30)
    at emitNone (events.js:105:13)
    at DestroyableTransform.emit (events.js:207:7)
    at prefinish (C:\Users\thoma\Documents\GitHub\test\node_modules\readable-stream\lib\_stream_writable.js:596:14)
    at finishMaybe (C:\Users\thoma\Documents\GitHub\test\node_modules\readable-stream\lib\_stream_writable.js:604:5)
    at afterWrite (C:\Users\thoma\Documents\GitHub\test\node_modules\readable-stream\lib\_stream_writable.js:470:3)
    at _combinedTickCallback (internal/process/next_tick.js:144:20) 

And the stream works fine when not forcefully closed, so it isn't related to the ndjson file. Is it possible to close the stream in the middle of the request?

Answer 1

There are several solutions :

1. Send a header to close the connection : this.set("Connection", "close")
2. Ending stream thanks to this.end()

I do not know which one is the best.

So, in your example, you have an issue due to this (I recommand you to take a look to this link). Try the following code :

res
  .pipe(ndjson.parse())
  .on('data', obj => {
    rows.push(obj);
    if (rows.length === amount) {
      this.end();
    }
  })
  .on('end', () => {
    resolve(rows);
  });