Where to define Celery subtask queues

Question

I have a pluggable app I'm developing for a Django system. In it, I have a task for creating notifications that looks something like so:

installable_app.tasks

@app.task(name='tasks.generate_notifications')
def generate_notifications(...):

I have a pluggable app I'm developing for a Django system. In it, I have a task for creating notifications that looks something like so:

installable_app.tasks

@app.task(name='tasks.generate_notifications')
def generate_notifications(...):
  clients = get_list_of_clients()
  for client in clients:
    client_generate_notification.delay(client['name'], client['id'])
  return "Notification Generation Complete"

@app.task
def client_generate_notification(client_name, client_id):
  ...
  return result

Now I want this to run periodically which can be accomplished with Celery Beat using settings. I also want it to be on its own queue:

settings.py:

CELERYBEAT_SCHEDULE ={
  {'generate_schedule_notifications': {
     'task': 'tasks.generate_notifications',
     'schedule': crontab(hour=6, minute=0),
     'options': {'queue': 'notification_gen'},
     'args': ('schedule', 'Equipment', 'HVAC')},
  }
}

The first task, generate_notifications is run correctly on the queue notification_gen but the client_generate_notification subtasks are run on the default queue.

I know I can specify the queues specifically in the @task decorator but since this is a django app, I would rather they be specified where it is actually run.

I've looked into using the CELERY_ROUTES option but when I tried it, it seemed to overwrite the queues for other tasks I was running.

Is the best practice to define all the possible queues in CELERY_ROUTES or is there a better way to set up my task so that they will both run on the same queue?

Answer 1

Do you want something like it?