Mechanisms for creating long-running process

Question

Are there alternative mechanism(s) for creating a long-running process besides running an infinite loop?

The common pattern seems to be this:

while True:
    # Check for some condition or waiting for some event
    # Do some processing
    time.sleep(0.01)

I am particularly interested in the scenario where the process acts as a worker that listens to some event (e.g. waiting on a task queue).

What are the performance characteristics of the alternative approaches?

Answer 1

Prior art on "wait for and process job" thing has been done a few different ways:

• Worker processes or threads (See multiprocessing and threading for some helpful primitives)
• Event-based processing (asyncio, twisted, and a few others). Your asyncronous IO library raises an event when you get data on STDIN or whatever pipe you choose.
• Single-threaded with IO buffer. Depending on your desired load characteristics, it could be reasonable for a worker processes to just wait on the IO and process it when it comes. No fancy queueing. Just let the kernel buffer the IO and the calling process block when it gets full.

Answer 2

If you're anticipating the need for performance for a lot of long-running tasks (and scaling over more machines) , a more high-level system like Celery might be fit:

"Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling."