How to run parallel process with limit?

Question

How to run in Nodejs parallel process but limit the number of process that exist in the current bulk?

Example, I have an Array with 200 items ['word.doc', 'foo.pdf', 'a.txt', ...] and I need to run Work process for each item like so:

Work.exe word.doc

Work.exe foo.pdf

Work.exe a.txt

Work.exe ....

What I did is:

• forEach in the Array..
• call to exec from child_process lib.

But I want only 5 process each time. When some process will end a new item should be up and running. So every time I have only 5 process until all process are completed.

Not sure how it can be done.

Answer 1

The child_process lib as a "close" event

you could create a counter and add a listener to this event. Each time you call exec, you increment the counter. If your number of processes goes beyond a threshold (here, 5) you do not call the function anymore. (you could just wrap the increment and the exec call in one function and call it multiple times)

And when you receive a "close" event you could decrement the counter and if the counter hits 0, recall the function to spawn a child process N times.

You would have to keep a global variable for the array index though. hope this helps.

Answer 2

exec accepts a callback that will be invoked with the child process exits. You can easily combine this with async eachLimit:

const async = require('async');
const child_process = require('child_process');

// items = ['word.doc', 'foo.pdf', 'a.txt', ...]

async.eachLimit(items, 5, (item, done) => {
    child_process.exec(`Work.exe ${item}`, done);
});