Multiple queue listeners will run the same job over multiple processes

Question

I have a simple web application that is written with the Laravel 4.2 framework. I have configured the Laravel queue component to add new queue items to a locally running beastalkd server.

Essentially, there is a POST route that will add an item to the beanstalkd tube.

I then have supervisord set up to run artisan queue:listen as three separate processes. The issue that I am seeing is that the different queue:listen processes will end up spawning anywhere between one to three queue:worker processes for just one inserted job.

The end result being that one job inserted into the queue is sometimes being processed by multiple workers at the same time, something I am obviously trying to avoid.

The job code is relatively simple:

<?php

use App\Repositories\URLRepository;

class ProcessDataJob {
    private $urls;

    public function __construct(URLRepository $urls)
    {
        $this->urls = $urls;
    }

    public function fire($job, $data)
    {
        $input = $data['post'];

        if (!isset($data['post']) && !is_array($data['post'])) {
            Log::error('[Job #'.$job->getJobId().'] $input was empty inside CreateAuditJob. Deleting job. Quitting. Bye!');
            $job->delete();
            return false;
        }

        //
        // ... code that will take a few hours to run.
        //

        $job->delete();
        Log::info('[Job #'.$job->getJobId().'] ProcessDataJob was successful, deleting the job!');
        return true;
    }
}

The fun part being that most of the (duplicated) queue workers fail when deleting the job with this left in the errorlog:

exception 'Pheanstalk_Exception_ServerException' with message 'Job 3248 NOT_FOUND: does not exist or is not reserved by client'

The ttr (Time to Run) is set to 172,800 seconds (or 48 hours), which is much larger then the time it would take for the job to complete.

Answer 1

what's the job time_to_run when queued? If running the job takes longer than time_to_run seconds, the job is automatically re-queued and becomes eligible to be run by the next worker.

A reserved job has time_to_run seconds to be deleted, released or touched. Calling touch() restarts the timeout timer, so workers can use it to give themselves more time to finish. Or use a large enough value when queueing.

I've found the beanstalkd protocol document helpful https://github.com/kr/beanstalkd/blob/master/doc/protocol.md

Answer 2

Since you are running with Laravel, verify your queue.php configuration file.

Change the ttr value from 60 (default) to something else that is better for you.

    'beanstalkd' => array(
        'driver' => 'beanstalkd',
        'host'   => 'localhost',
        'queue'  => 'default',
        'ttr'    => 600, //Example
    ),