Celery rate_limit not respected when scheduling via 'eta' option

Question

I'd like to schedule a task to run at various times in the future. I'm trying to use the eta option when calling the task to do this. This part works fine on its own, but at the same time, I'd like to have a rate limit on the task. I'm not able to do both: schedule the task to run in the future, but also get celery to respect the rate limit.

Here is an example (rate_limit_eta.py)

from celery import Celery
from datetime import datetime, timedelta

app = Celery('rate_limit_eta', broker='amqp://guest@localhost//')

@app.task(rate_limit='4/m')
def my_task():
    print 'execution timestamp: {}'.format(datetime.utcnow())


if __name__ == '__main__':

    now = datetime.utcnow()
    d = now + timedelta(seconds=5)
    print now

    for i in range(10):
        my_task.apply_async(eta=d)

In the loop, when I call 'my_task' with eta=d, the tasks get executed as fast as possible. The rate_limit is not respected.

my_task.apply_async(eta=d)


[2015-06-12 13:45:15,750: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.748498
[2015-06-12 13:45:15,757: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.757757
[2015-06-12 13:45:15,760: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.759992
[2015-06-12 13:45:15,763: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.763614
[2015-06-12 13:45:15,766: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.765996
[2015-06-12 13:45:15,768: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.768424
[2015-06-12 13:45:15,771: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.771079
[2015-06-12 13:45:15,774: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.774561
[2015-06-12 13:45:15,777: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.777425
[2015-06-12 13:45:15,780: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.780207

But if I do not schedule the task (i.e. without the eta), then the rate limit is enforced.

my_task.apply_async()

[2015-06-12 13:47:05,844: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.843872
[2015-06-12 13:47:05,850: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.850502
[2015-06-12 13:47:21,919: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:21.919120
[2015-06-12 13:47:35,937: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:35.937060
[2015-06-12 13:47:51,959: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:51.959174
[2015-06-12 13:48:05,976: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:05.976197
[2015-06-12 13:48:21,998: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:21.998254
[2015-06-12 13:48:36,014: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:36.014837
[2015-06-12 13:48:52,041: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:52.041326
[2015-06-12 13:49:06,057: WARNING/Worker-1] execution timestamp: 2015-06-12 13:49:06.057286

Am I doing something wrong?

Answer 1

I had the same issue, and I found why. I had a look into Celery code and this behaviour is coming from the strategy code, which does something along the lines of:

if req.eta:
    apply_eta()
else:
    check_if_rate_limited()

I'm afraid it's something to do that ETAs and rate limit being 2 incompatible ways to tell Celery when the task should be executed. Celery has to choose one over the other, and it happens to be ETA.

My link is to v3 of Celery (as I suspect this question was asked for v3), but at time of writing, this is still valid on Celery's master branch. I guess this limitation could be documented a bit better somewhere (here?), it's just a pull request away!

UPDATE: I've opened an issue on the Celery project.