Rails: Batch Subscribe users to MailChimp API on Heroku running into timeout issues

Question

Here is my code

  users = User.all

  # Latency issues with connecting with Heroku and MC
  Gibbon::API.timeout = 120

  gb = Gibbon::API.new
  batch = []
  users.each do |user|
    batch << user.mail_chimp_information
  end

  puts gb.lists.batchSubscribe(id: "MC_ID_HERE", batch: batch, double_optin: false, update_existing: true)

The code above is set to run on a nightly cron that batch subscribes (or updates existing) users to my MailChimp account. My app is running on Heroku which causes issues with retrieving users and then looping through them before sending them to MailChimp. If I remove the Gibbon::API.timeout = 120 line, then the default is 15 seconds and times out.

What is the best practice on batch uploading of user information to an external API? Manually setting the Timeout is a quick fix for now, but as my user base grows, the greater the threat for time outs to occur again.

Answer 1

Best practice is to use a delayed job to do the work for you.

The idea is that you setup each job as a one off (instead of many) delayed job and then have a background queue that handles the subscribing.

The common gems are:

• DelayedJob
• Resque
• Sidekiq