How can I get rid of legacy tasks still in the Celery / RabbitMQ queue?

Question

I am running Django + Celery + RabbitMQ. After modifying some task names I started getting "unregistered task" KeyErrors, even after removing tasks with this key from the Periodic tasks table in Django Celery Beat and restarting the Celery worker. They persist even after running with the --purge option.

How can I get rid of them?

Answer 1

To flush out the last of these tasks, you can re-implement them with their old method headers, but no logic.

For example, if you removed the method original and are now getting the error

[ERROR/MainProcess] Received unregistered task of type u'myapp.tasks.original'

Just recreate the original method as follows:

tasks.py

@shared_task
def original():
    # keep legacy task header so that it is flushed out of queue
    # FIXME: this will be removed in the next release
    pass

Once you have run this version in each environment, any remaining tasks will be processed (and do nothing). Ensure that you have removed them from your Periodic tasks table, and that they are no longer being invoked. You can then remove the method before your next deployment, and the issue should no recur.

This is still a workaround, and it would be preferable to be able to review and delete the tasks individually.