Throttling HTTP request (Node.js)

824 views Asked by At

I'm trying to do some basic web scraping and eventually found the need to limit my request calls as the server will return a page not found when there's too many request

Currently I'm using request-promise wrapped in request-promise-retry to make this call and I also found this article which seems to be trying to achieve the same thing Throttle and queue up API requests due to per second cap

I went ahead to try the simple-rate-limiter as it looks simple enough to use but got the following error

TypeError: requestPromise(...).catch is not a function
    at promiseRetry.retries (C:\project\node_modules\request-promise-retry\index.js:18:27)
    at C:\project\node_modules\promise-retry\index.js:29:24
    at <anonymous>

My guess is that the simple-rate-limiter doesn't work with request-promise and only works with request.

Are there any simple ways to go around throttling the requests without having to rewrite all my calls with "request" instead of "request-promise"?

Or is it a good idea to rewrite using normal "request", which will be a pain as I already have a few pieces of code written with request-promise and is expecting a promise to be returned.

0

There are 0 answers