There are a lot of documents in my mongodb, that need to be processed at 11 pm and as soon as they are processed, they need to be updated. So, a cron is written that fetches these documents by date and time. But, to scale up, there will be thousands of documents that will need to process. We thought of splitting the cron into 4 smaller crons. But, here the problem was that if 2000 documents are needed to processed and updated in a day, we have to process them 500 at a time, but where do we store the value 500 for one day ? Is it efficient to split that cron? if it is efficient, how can I split them ? P.S. While processing can be done in parallel threads, updating the documents should be done in sync because of read-write conflict of database.

0 Answers