My project currently has about 300 scheduled tasks. Some of these run every hour on the hour and some run once a day (at midnight). The tasks make API calls to third party API's.
This is the current set up of my Q_CLUSTER
Q_CLUSTER = {
'name': 'DataScheduler',
'workers': 3,
'timeout': 15,
'retry': 300,
'queue_limit': 10000,
'bulk': 10,
'max_attempts': 1,
'orm': 'default'
}
Of the 300 tasks, about 120 of them fail (different ones every time). If I manually trigger these to run they pass so there isn't anything wrong with the actual request. I believe the q cluster can't process all of the tasks when they are triggered at the same time. I think this can be improved by tweaking the Q_CLUSTER settings but I'm unsure how best to do this.
I've tried running through the documentation https://django-q.readthedocs.io/en/latest/configure.html
But I'm unsure how to improve the success rates of the scheduled tasks. I've struggled to find any other articles or documentation to explain how best to utilise these settings.
My question is - how can I improve these settings to ensure the tasks pass? With time I will have more scheduled tasks so looking to address now before it gets worse.
Appreciate any advice or help in the right direction for where to go on this.