Why does Celery task autodiscovery never trigger under Django?

Question

Everything is working except task autodiscovery under Django. (Celery 5.2.6, Django 3.0.6)

If I run celery worker, the worker triggers autodiscovery, finds all my tasks, and displays them in a list as part of it's startup process. However, if I run my Django app or Django shell, this never happens.

Additionally, even though the docs promise that accessing the task registry will trigger autodiscovery, it does not. I print app.tasks, I call app.tasks.keys(), and still no autodiscovery - it only shows the tasks that are built-in or were registered when the module containing them happened to be imported for other reasons.

What do I need to do to trigger task autodiscovery?

PS - If I try adding force=True to app.autodiscover_tasks(), it fails because the Django app registry hasn't finished loading at that time.

Answer 1

I dug into the code and tried a bunch of things. Here's what I learned.

How Autodiscovery is Performed

Autodiscovery is actually performed by app._autodiscover_tasks() (src), which is invoked by the celery.signals.import_modules signal (src), which is sent by app.loader.import_default_modules() (src). The signal is connected to app._autodiscover_tasks() in app.autodiscover_tasks (src).

In other words, the only way to get autodiscovery to trigger is to:

1. call app.autodiscover_tasks() to register app._autodiscover_tasks as a receiver for signal celery.signals.import_modules
2. call app.loader.import_default_modules() to send the signal

The docs instruct me to do the first, but not the second. Thus, in my Django app, autodiscovery does not happen.

How Autodiscovery is Performed in celery

If you run celery with the commands beat, report, shell, or worker, the resulting code path eventually calls app.loader.import_default_modules() directly. Thus, autodiscovery happens under those conditions - but not when you run your Django app, or open a Django shell, or any other context but those four commands, unless you explicitly call app.loader.import_default_modules().

On Finalize

The docs promise that accessing app.tasks will cause the app to finalize, which I assumed triggered autodiscovery, but it has nothing to do with that. All that finalize does is send an app.on_after_finalize signal, which does nothing.

My Solution

After much frustrating and experimentation, I settled on this snippet from my celery.py:

app = Celery("myproj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

@app.on_after_configure.connect()
def trigger_autodiscovery(sender, **kwargs):
    app.loader.import_default_modules()