Getting jobs from beanstalkd - timed out exception

Question

I am using Python 2.7, beanstalkd server with beanstalkc as the client library.

It takes about 500 to 1500 ms to process each job, depending on the size of the job.

I have a cron job that will keep adding jobs to the beanstalkd queue and a "worker" that will run in an infinite loop getting jobs and processing them.

eg:

def get_job(self):
    while True:
        job = self.beanstalk.reserve(timeout=0)
        if job is None:
            timeout = 10 #seconds
            continue
        else:
            timeout = 0 #seconds
            self.process_job(job)

This results in "timed out" exception.

Is this the best practice to pull a job from the queue?

Could someone please help me out here?

Answer 1

Calling beanstalk.reserve(timeout=0) means to wait 0 seconds for a job to become available, so it'll time out immediately unless a job is already in the queue when it's called. If you want it never to time out, use timeout=None (or omit the timeout parameter, since None is the default).