NodeJS Worker as slow/slower than as regular Single Thread Program

27 views Asked by At

Before anything, I know that I WANT to make my own database system, I already made one in the past that worked perfectly, I am simply looking to take on difficult challenges.

This Database system will be memory based. I am looking for methods to read / write data the most efficient way and for that I made a couple of benchmarks, here are the code/results:

// 1. JSON array system | 84.289ms
let testDataJson = new Array(10000000).fill({ id: 1, username: "TestUsername", password: "MyPassword1234", email : "[email protected]" })
testDataJson.push({ username: "FoundUsername" }); // Insert matching element

console.time("dataJson");
console.log(testDataJson.find(elem => elem.username === "FoundUsername");
console.timeEnd("dataJson");
// 2. TEXT array system | 86.041ms
let testDataText = new Array(10000000).fill(`1|TestUsername|MyPassword1234|[email protected]`.split("|"));
testDataText.push(`|FoundUsername||`.split("|"));

console.time("dataText");
console.log(testDataText.find(elem => elem[1] === "FoundUsername");
console.timeEnd("dataText");

Now onto the problematic part, I decided for this version to work using NodeJS Workers to spread the load accross multiples threads. For that I am spreading the array into x amount of workers and split the data array. I am then loading in each worker memory its designated data block and wait 3 seconds to let everything settle. I then run the search, and the result is, quite bad, here is my code


// PROGRAM CODE | 56.698ms
function loadThreads()
{
    let dataChunks = fs.readFileSync("./db.txt", 'utf-8').split("\n");
    let workersParts = chunkArrayInGroups(dataChunks, 2000000); // File is 10 000 000 Lines, resulting in 5 workers
    for (let workerPart of workersParts)
    {
        const worker = new Worker("./dbTester_thread.js", { workerData: workerPart.join("\n") });
        threads.push(worker);
    }
}

function performSearch()
{
    console.time("threadTest");
    for (let thread of threads)
    {
        thread.on('message', (message) => {
            console.timeEnd("threadTest")
        });
        thread.postMessage("HelloMan")
    }

}

// THREAD CODE
const { parentPort, workerData } = require('node:worker_threads');

let data = workerData.split("\n").map(line => line.split(","));
parentPort.on('message', (message) => 
{
  let found = data.find(line => line[1] === message);
  if (found) parentPort.postMessage("FOUND");
});

My first theory was that maybe parentPort.postMessage is poorly optimised and takes x amount of time to fully contact the main thread, but after trying an immediate postMessage after any message reception, the delay of an instant response was 0.444ms so I am out of explainations.

I am out of solutions besides what I placed up there

0

There are 0 answers