Infrastructure for Running your Zend Queue Receiver

Question

I have a simple messaging queue setup and running using the Zend_Queue object heirarchy. I'm using a Zend_Queue_Adapter_Db back-end. I'm interested in using this as a job queue, to schedule things for processing at a later time. They're jobs that don't need to happen immediately, but should happen sooner rather than later.

Is there a best-practices/standard way to setup your infrastructure to run jobs? I understand the code for receiving a message from the queue, but what's not so clear to me is how run the program that does that receiving. A cron that receives n messages on the command-line, run once a minute? A cron that fires off multiple web requests, each web request running the receiver script? Something else?

Tangential bonus question. If I'm running other queries with Zend_Db, will the message queue queries be considered part of that transaction?

Answer 1

You can do it like a thread pool. Create a command line php script to handle the receiving. It should be started by a shell script that automatically restarts the process if it dies. The shell script should not start the process if it is already running (use a $pid.running file or similar). Have cron run several of these every 1-10 minutes. That should handle the receiving nicely.

I wouldn't have the cron fire a web request unless your cron is on another server for some strange reason.

Another way to use this would be to have some backround process creating data, and a web user(s) consume it as they naturally browse the site. A report generator might work this way. Company-wide reports are available to all users but you don't want them all generating this db/time intensive report. So you create a queue and process one at a time possible removing duplicates. All users can view the report(s) when ready.

According to the docs it doens't look like the zend db is even using the same connection as your other zend_db queries. But of course the best way to find out is to make a simple test.

EDIT The multiple lines in the cron are for concurrency. each line represents a worker for the pool. I was not clear, you don't want the pid as the identifier, you want to pass that as a parameter.

• • • • • /home/byron/run_queue.sh Process1
• • • • • /home/byron/run_queue.sh Process2
• • • • • /home/byron/run_queue.sh Process3

The bash script would check for the $process.running file if it finds it exit.

otherwise:

• Create the $process.running file.
• start the php process. Block/wait until finished.
• Delete the $process.running file.

This allows for the php script to die but not cause the pool to loose a worker.

If the queue is empty the php script exits immediately and is started again by the nex invocation of cron.