How to Wait for Parallel Sidekiq Workers to All Complete Before Calling Another Method?

Question

I have several workers that are being run using Sidekiq and scheduled using Sidetiq. I'm looking for advice on the best way to wait for all workers to complete before executing a callback, similar to Sidekiq-Pro's batching functionality. Any advice on possible options would be greatly appreciated!

Answer 1

For simple use cases you can poll stats to get a count of pending jobs for each queue.

So if you know that all your jobs go to a specific queue, and there are no other jobs going in that queue from elsewhere this would suffice. For more advanced/complex use cases you can refer the API source

Another simple solution is to use a Redis based counter (since you are already using Redis) and atomically decrement it from each job, and publish an event once the count reaches zero.

Answer 2

You can write a method:

def wait_for_sidekiq
  sleep(1) until Sidekiq::Workers.new.size == 0 && Sidekiq::Queue.new.size == 0
end

I would also suggest that you make sure that the jobs got queued in the first place:

def wait_for_queuing
  sleep(1) until Sidekiq::Queue.new.size > 0 || Sidekiq::Workers.new.size > 0
end

This is required, because sometimes the first method might get executed a few miliseconds before queuing the jobs, so that it won't wait at all.

Answer 3

You should see this Sidekiq-batch gem, i already try it and it works just like the Sidekiq's Pro Batch, and it's free

Hope this could help