Reducing Node memory usage when making HTTP requests in a loop

781 views Asked by At

I've set up a simple loop to poll an IronMQ messaging system, and everything works fine... except that memory usage increases more and more until it finally stabilizes at over 250MB. I've read that it's normal for Node to use more memory over time when run in a (sort of) recursive loop like this, even when running setTimeout and doing nothing, but I still don't understand the exact mechanics behind this behavior, or whether there is any way to control it. When making HTTP requests within the loop, memory usage more than doubles.

The code is running on a Heroku worker with a limit of 512MB RAM, leaving no breathing room to use cluster to use the rest of the available CPU cores. The memory usage can increase slowly or extremely quickly, depending on the jobs that run after receiving the messages.

This is the simplest code that reproduces this.

(function loop() {
  request.get('http://www.example.com', function(err, request, body) {
    if (err) console.log(err);

    setTimeout(loop, 200);
  });
})();

I've tried many, many ways of restructuring this code to prevent memory from increasing so high, but nothing has made any changes. Only the received HTTP response seems to have any effect on the upper limit of RAM used.

Is there a way to rewrite this entirely, or am I stuck with V8's behavior? All examples I've found use the same basic structure for infinite async loops, from kue to the async library.

0

There are 0 answers