I am using delayed_jobs that for some of them have dependencies that need to be executed after the current one and with a small delay. This delay comes from rate limitting of the API I am querying.
Basic idea
Here are my jobs constraints:
1s 1s
A1 --------> A2 --------> A3
and
1s 1s
B1 --------> B2 --------> B3
In the end, I would like this process to take ~2s with something like:
1s 1s
A1 -> B1 -----> B2 -> A2 ------> A3 -> B3
or
1s 1s
A1 -> B1 -----> A2 -> B2 ------> A3 -> B3
or whatever order which respects my first constraints, but with dependencies and waiting.
More context
I am currently building an app for Shopify. An app for Shopify is rate limitted by 2 API calls/s per shop.
But the webhooks coming from shopify that send me jobs to do can be way more than 2 per second per shop so I need to use jobs to handle them.
Then each job actually send two request to their API.
Considered approach
I could use a second queue and use the after hook to fetch the next job to do for the shop and add it to the jobs table with run_at: 1.seconds.from_now. However, I don't really like the idea, because I'll have to store the perform arguments in the table instead of just calling a perform_later method. Also, I really imagine that this isn't the first use case where dependencies are needed with delayed_job, so I'm sure there is a better approach to it that I just couldn't find.
Any idea would really be appreciated!
I am not at all clear what you're trying to do, but a good common pattern that has worked for me most of the time (many millions of times with a few exceptions)...
when you get a webhook call, create the job (if you have not already seen that webhook call), and return 200 status as fast as possible. That keeps Shopify happy and stops them sending you the same hook over again.
MonkeyPatch ActiveResource to inspect the credit left header number in your App, so that you can easily go into a 500ms wait if the calls left are zero. Works a charm... for 2/second limit.