Celery prefetching not working celery app argument

Question

I have a long task that I am asynchronously running with celery. Since I am using multiple workers I was getting poor load balancing due to prefetching. I initially tried to turn it off like this

# __init__.py 
from celery import Celery

celery = Celery(
                __name__,
                broker=CONFIG['CELERY_BROKER_URL'],
                backend=CONFIG['CELERY_BROKER_URL'], 
                worker_prefetch_multiplier=1,
                #task_acks_late=True,
            )

# celery_runner.py
from server import celery
from server.app import create_app, init_celery
app = create_app()

init_celery(app, celery)
from . import tasks

And running my worker using celery -A server.celery_runner.celery worker --loglevel=info -c 1 -n w1 -Ofair -E

But it did not give any result. On searching, I found this S/O post 37351549 which suggests the command-line option.

On switching to celery -A server.celery_runner.celery worker --loglevel=info -c 1 -n w1 -Ofair -E --prefetch-multiplier 1 which worked. Why did it not work through the celery init option?

Answer 1

I have never passed config parameters directly, as arguments to the Celery constructor. Instead I always used the super-useful config_from_object() method. I suggest you give it a try.