How can I reverse a stream to read a JSONL backward?

27 views Asked by At

I have a JSONL online file I need to parse line by line in the following format:

{ "id": "something1" }
{ "id": "foo1", "__parentId": "something1" }
{ "id": "something2" }
{ "id": "something3" }
{ "id": "foo2", "__parentId": "something3" }
{ "id": "something4" }
{ "id": "foo3", "__parentId": "something2" }

This file can be quite large (even 20GB). Currently I have the following code:

await axios.get(url, { responseType: "stream" }).then(async (res) => {
  const rl = readline.createInterface({
    input: res.data,
    crlfDelay: Infinity,
  });

  for await (const line of rl) {
    if (line) {
      await handleLine(JSON.parse(line));
    }
  }
});

This code works, but it will read the file from top to bottom, I need to parse it from bottom to top, because in this way once I process a parent I know all his children have already been processed.

I tried with saving it in a tmp file and read the file backward, this worked but as I said the file can potentially be infinite large, so I would like to find something to handle it directly with the nodejs streams, without saving it in my machine memory. I know streams are designed to be read sequentially from start to end, but is there a way to transform it, maybe with TransformStream?

0

There are 0 answers