I'm trying to do some basic web scraping and eventually found the need to limit my request calls as the server will return a page not found when there's too many request
Currently I'm using request-promise wrapped in request-promise-retry to make this call and I also found this article which seems to be trying to achieve the same thing Throttle and queue up API requests due to per second cap
I went ahead to try the simple-rate-limiter as it looks simple enough to use but got the following error
TypeError: requestPromise(...).catch is not a function
at promiseRetry.retries (C:\project\node_modules\request-promise-retry\index.js:18:27)
at C:\project\node_modules\promise-retry\index.js:29:24
at <anonymous>
My guess is that the simple-rate-limiter doesn't work with request-promise and only works with request.
Are there any simple ways to go around throttling the requests without having to rewrite all my calls with "request" instead of "request-promise"?
Or is it a good idea to rewrite using normal "request", which will be a pain as I already have a few pieces of code written with request-promise and is expecting a promise to be returned.