lowDb - out of memory

Asked by At

I have an Out-Of-Memory Problem in Node.js and see a lot of big strings that can't be garbage collected when I inspect the snapshots of the heap.

I use lowDB and those strings are mainly the content of the lowDb file.

Question in principle...

When I use FileAsync (so the writing to the file is asynchronous) and I do a lot of (fire and forget) writes...is it possible that my heap space is full of waiting stack entries that all wait for the file system to finish writing? (and node can clear the memory for each finished write).

I do a lot of writes as I use lowDB to save log messages of an algorithm that I execute. Later on I want to find the log messages of a specific execution. So basically:

{ 
  executions: [
    {
      id: 1,
      logEvents: [...]
    },
    {
      id: 2,
      logEvents: [...]
    },
    ...       
  ]
}

My simplified picture of node processing this is:

  • my script is the next script on the stack and runs
  • with each write something is waiting for the file system to return an answer
  • this something is bloating my memory and each of this 'somethings' hold the whole content of the lowdb file (multiple times?!)

Example typescript code to try it out:

import * as lowDb from 'lowdb';
import * as FileAsync from 'lowdb/adapters/FileAsync';

/* first block just for generating random data... */
const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
const charactersLength = characters.length;
const alphanum = (length: number) => {
    const result = new Buffer(length);
    for (let i = 0; i < length; i++ ) {
        result.write(characters.charAt(Math.floor(Math.random() * charactersLength)));
    }
    return result.toString('utf8');
};

class TestLowDb {
    private adapter = new FileAsync('test.json');
    private db;

    /* starting the db up, loading with Async FileAdapter */
    async startDb(): Promise<void> {
        return lowDb(this.adapter).then(db => {
            this.db = db;
            return this.db.defaults({executions: [], dbCreated: new Date()}).write().then(_ => {
                console.log('finished with intialization');
            })
        });
    }

    /* fill the database with data, fails quite quickly, finally produces a json like the following:
    * { "executions": [ { "id": "<ID>", "data": [ <data>, <data>, ... ] }, <nextItem>, ... ] } */
    async fill(): Promise<void> {
        for (let i = 0; i < 100; i++) {
            const id = alphanum(3);
            this.start(id); // add the root id for this "execution"
            for (let j = 0; j < 100; j++) {
                this.fireAndForget(id, alphanum(1000));
                // await this.wait(id, alphanum(1000));
            }
        }
    }

    /* for the first item in the list add the id with the empty array */
    start(id:string): void {
        this.db.get('executions')
            .push({id, data:[]})
            .write();
    }

    /* ignores the promise and continues to work */
    fireAndForget(id:string, data:string): void {
        this.db.get('executions')
            .find({id})
            .get('data')
            .push(data)
            .write();
    }

    /* returns the promise that the caller can handle it "properly" */
    async wait(id:string, data:string): Promise<void> {
        return this.db.get('executions')
            .find({id})
            .get('data')
            .push(data)
            .write();
    }
}

const instance = new TestLowDb();
instance.startDb().then(_ => {
    instance.fill()
});
enter code here

0 Answers