The requirements are distributed task processing and programming tasks in Python for a high message rate. How do Celery and Storm (with streamparse) compare on the following tenets:
Scalability- not only in terms of workers, but also in context of brokers. Can Celery work with horizontally scalable brokers (sharded rabbitmq clusters)? Does Storm have any such limitations?
Flexibility with tasks- If a task must for a specific condition in the system, it may well be re-queued and processed when it is next absorbed by a worker. Does Storm provide such features?
Workflows- Workflows tend to get complicated- some sequentially, some in parallel and then their combinations. Which is better here?
Monitoring- Real-time monitoring support for workers, their status, their message rates. Alerting features in case of errors.
Ease of deployment