Rails - passing data from background job to main thread

Question

I am using in my app a background job system (Sidekiq) to manage some heavy job that should not block the UI.

I would like to transmit data from the background job to the main thread when the job is finished, e.g. the status of the job or the data done by the job.

At this moment I use Redis as middleware between the main thread and the background jobs. It store data, status,... of the background jobs so the main thread can read what it happens behind.

My question is: is this a good practice to manage data between the scheduled job and the main thread (using Redis or a key-value cache)? There are others procedures? Which is best and why?

Answer 1

Here are two other options that I have used in the past:

• Unix sockets. This was extremely fiddly, creating and closing connections was a nuisance, but it does work. Also dealing with cleaning up sockets and interacting with the file system is a bit involved. Would not recommend.
• Standard RDBMS. This is very easy to implement, and made sense for my use case, since the heavy job was associated with a specific model, so the status of the process could be stored in columns on that table. It also means that you only have one store to worry about in terms of consistency.

I have used memcached aswell, which does the same thing as Redis, here's a discussion comparing their features if you're interested. I found this to work well.

If Redis is working for you then I would stick with it. As far as I can see it is a reasonable solution to this problem. The only things that might cause issues are generating unique keys (probably not that hard), and also making sure that unused cache entries are cleaned up.

Answer 2

Redis pub/sub are thing you are looking for. You just subscribe main thread using subscribe command on channel, in which worker will announce job status using publish command. As you already have Redis inside your environment, you don't need anything else to start.