I am working on a Flask app and using APScheduler to schedule jobs and Celery to run that jobs as background processes. When time comes for a particular task, APScheduler assigns that task to Celery, but sometimes Celery worker does not execute that task until Celery app is restarted.
def run_job(data)
logger.info('Queuing data send')
# Call celery task to send post_data to URL
send_request.apply_async(data)
where send_request() is a Celery task. So, I get logs for "Queuing data send" for multiple jobs but execution never reaches inside send_request method unless Celery app is restarted.
I am running Celery app as
celery_app.start(argv=['celery', 'worker', '-Ofair', '--without-gossip', '--without-mingle', '-l', 'info', '--autoscale', '12,3', '-Q', 'my_queue', '-n', 'my_queue'])