Bull queue: Ensure unique job within time period by using partial timestamp in jobId

Question

I need to ensure the same job added to queue isn't duplicated within a certain period of time.

Is it worth including partial timestamps (i.e. D/M/Y-HH:M) in my unique jobId strings, so it processes only if not in the same Minute?

It would still duplicate if one job was added at 12:01 and the other at 12:09 – or does Bull have a much better way of doing this?

Answer 1

I feel you should use Bull's API to check that the job is running or not, then you decide if you add the job to the queue if not (patch on the producer).

You can also decide to check if a similar job is already running when your are running the job (inside the process function) and do an early return instead of executing the job (patch on the consumer).

You can use the Queue getJobs function to do so:

getJobs(types: string[], start?: number, end?: number, asc?: boolean):Promise<Job[]>

"Returns a promise that will return an array of job instances of the given types. Optional parameters for range and ordering are provided."

From documentation: https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queuegetjobs

The Job item should provide enough data so you can find the one you are looking for.

Answer 2

Bull is designed to support idempotence by ignoring jobs that were added with existing job ids. Be careful to not enable options such as removeOnCompleted, since the job will be removed after completion and not being considered the next time you add a job.

In your case, where you want to make sure that no new jobs are added during a given timespan, just make sure that all the job ids during that timestamp are the same, for example as you wrote in your comment removing the 4 last digits of your UNIX timestamp.