Correcting mis-configured celery (running with supervisord)

Question

I started running celery for tasks in a Python/Django web project, hosted on a single VM with 8 cores or CPUs. I need to improve the configuration now - I've made rookie mistakes.

I use supervisor to handle celery workers and beat. In /etc/supervisor/conf.d/, I have two worker-related conf files - celery1.conf and celery1.conf. Should I...

1) Remove one of them? Both spawn different workers. I.e. the former conf file has command=python manage.py celery worker -l info -n celeryworker1. The latter has command=python manage.py celery worker -l info -n celeryworker2. And it's authoritatively stated here to run 1 worker per machine.

2) Tinker with numprocs in the conf? Currently in celery1.conf, I've defined numprocs=2. In celery2.conf, I've defined numprocs=3* (see footer later). At the same time, in /etc/default/celeryd, I have CELERYD_OPTS="--time-limit=300 --concurrency=8". So what's going on? supervisor's numprocs takes precedence over concurrency in celeryd, or what? Should I set numprocs=0?

---

*total numprocs over both files = 2+3 = 5. This checks out. sudo supervisorctl shows 5 celery worker processes. But in newrelic, I see 45 processes running for celeryd. What the heck?! Even if each proc created by supervisor is actually giving birth to 8 procs (via celeryd), total numprocs x concurrency = 5 x 8 = 40. That's 5 less than the 45 shown by newrelic. Need guidance in righting these wrongs.

Compare screenshots:

vs

Answer 1

it's authoritatively stated here to run 1 worker per machine

Actually, it's advised ("I would suggest") to only run one worker per machine for this given use case.

You may have pretty good reason to do otherwise (for example having different workers for different queues...), and the celery doc states that how many worker / how many processes per worker (concurrency) works best really depends on you tasks, usage, machine and whatnots.

wrt/ numprocs in supervisor conf and concurrency in celery, these are totally unrelated (well, almost...) things. A celery "worker" is actually one main process spawning concurrency children (which are the ones effectively handling your tasks). Supervisor's numprocs tells supervisor how many processes (here: the celery workers) it should launch. So if you have one celery conf with numprocs = 2 and another one with numproc = 3, this means you launch a total of 5 parents worker processes - each of them spawning n subchilds, where - by default - n is your server's cpus count. This means you have a total of 5 + (5*8) = 45 worker subprocesses running.

Wether you actually need that many workers is a question only you can answer ;)