Queued job in Laravel runs twice, but only intermittently

Question

I'm in charge of a reasonably mature Laravel 10 site and am having problems with nightly queued jobs getting run twice but only occasionally.

We're running on a single dedicated server, using Redis with Horizon for queue management.

Here's an example of the scheduled job, runs on Saturday morning at 12:30 AM and processes important jobs for the following Monday:

$schedule->call(function () {
   dispatch(new RunImportantDailyJob(Carbon::parse('monday')->format('Y-m-d')))->onQueue('long-running-queue');
})->weekly()->saturdays()->at('0:30')->timezone(nova_get_setting('store_timezone'));

19 times out of 20, it runs perfectly. But that one time, it runs twice. Here are the log entry from the very start of the job on a day when it ran twice:

[2024-01-27 00:30:03] production.INFO: Running Important Daily Job where next activity date <= 2024-01-29
[2024-01-27 00:30:05] production.INFO: Running Important Daily Job where next activity date <= 2024-01-29

No indication of any job processing failures in Horizon or the Laravel logs.

Here are my settings for long-running-queue in config/queue.php:

'redis-long-running' => [
   'driver' => 'redis',
   'connection' => 'default',
   'queue' => env('REDIS_QUEUE', 'default'),
   'retry_after' => 2400,
   'block_for' => null,
],

...and in config/horizon.php:

'supervisor-long-running' => [
   'connection' => 'redis-long-running',
   'queue' => [
      'long-running-queue',
   ],
   'balance' => 'simple',
   'processes' => 9,
   'tries' => 1,
   'timeout' => 2300,
],

I'm a long-time backend developer but have only been working with Laravel for about a year. Never had any problems with nightly cron runs at previous jobs, so this is frustrating.

I've updated the code in my Job to be as 'idempotent' as possible - i.e., checking that any critical transactions have not already been processed, but I really want to get to the bottom of this.

I've tried the 'withoutOverlapping' technique described in the Laravel docs, but every other week the job just wouldn't run at all, which was just as bad.

Perhaps I shouldn't even be queuing this job, and should run it as an artisan command instead? It only takes about 45 seconds to run.

Searching this problem, it seems that I'm not the only one having it, but I haven't found any applicable diagnostic approaches.

Any advice about how to solve this, or how to use my tools (Horizon, Telescope, other logs) to troubleshoot and figure out what's really going on here?

Thanks!

• Andrew