Is there a way to solve an ActiveStorage integrity error when doing post processing on an image after upload to Cloudinary?

Question

I am currently working on a Rails app using Rails (6.0.3.2). I am using ActiveStorage and the Cloudinary gem to upload the photos to Cloudinary.

Everything works fine when uploading the photos normally. However, when I set a rule on Cloudinary to resize photos upon upload my logs end up full of the following error. To be specific photos are resized and then saved to the cloud storage, replacing the original, not simply creating a variant. I believe this is where the issue lies.

2020-10-09T04:47:28.917859+00:00 app[web.1]: I, [2020-10-09T04:47:28.917748 #9]  INFO -- : 
[ActiveJob] Enqueued ActiveStorage::AnalyzeJob (Job ID: 09e91b28-ee03-48e8-b04f-7120a06910b8)
 to Async(active_storage_analysis) at 2020-10-09 04:47:46 UTC with arguments: #
<GlobalID:0x000055a832b56848 @uri=#<URI::GID gid://trace-taiwan/ActiveStorage::Blob/268>>

2020-10-09T04:47:28.918128+00:00 app[web.1]: I, [2020-10-09T04:47:28.918053 #9]  INFO -- : 
Retrying ActiveStorage::AnalyzeJob in 18 seconds, due to a ActiveStorage::IntegrityError.

This repeats after 18 seconds then 80 seconds then 256 seconds and so on. Beyond this error, everything actually works okay. The photos are uploaded, resized, and displayed via the app both in development and production, but I don't want this process running and failing like this constantly on every upload. Again if I don't resize the photos this goes away, however with the number of photos I will be allowing users to be uploading and my limited storage this isn't a great option for me.

Is there any way to let ActiveStorage know the photos have been and should have been resized, or perhaps a way to have this check run immediately or not at all?

Answer 1

So after extensive searching part of the information used to create the checksum for the integrity check is the 'byte-size' of the image.

The checksum is first created before passing an image to Cloudinary and then after it has successfully been saved. As service side image processing taking place on Cloudinary on incoming images resizes them before saving them there really isn't a way to do this.

As a workaround what I did is created a background job I can call after the post is created.

require 'tmpdir'
require 'fileutils'
require 'open-uri'
class ResizeImagesJob < ApplicationJob
  queue_as :default

  def perform(post)
    post.images.each do |image|
      blob = image.blob
      blob.open do |temp_file|
        path = temp_file.path
        pipeline = ImageProcessing::MiniMagick.source(path)
        .resize_to_limit(1200, 1200)
        .call(destination: path)
        new_data = File.binread(path)
        post.send(:images).attach io: StringIO.new(new_data), filename: blob.filename.to_s, 
                           content_type: 'image'
      end
      image.purge_later
    end
  end
end

After saving the post with the images attached I call the job like this

ResizeImagesJob.set(wait: 2.seconds).perform_later(@post)

What this does is uploads the photos to Cloudinary as normal so there is no extra wait noticed by the user like there would be if I resized them myself. Waits 2 seconds to allow the integrity check to take place before I change the photos. It then re-downloads the photos from Cloudinary, re-sizes them, attaches the newly resized photo to the post, and deletes the old one, all in the background.

Obviously, this isn't ideal, but it works. More than happy to hear any better solutions.

Answer 2

I got it working in a before_save hook with the following code

class Thing < ApplicationRecord
  has_many_attached :pictures

  before_save :resize_pictures

  private

  # Note: AbcSize: 21.47
  def resize_pictures
    # Loop through attached files
    pictures.each_index do |index|
      # Ignore previously saved attachments
      next if pictures[index].persisted?

      # The real uploaded file (in my case, a Rack::Test::UploadedFile)
      # This is an undocumented Rails API, so it may change or break in future updates
      file = attachment_changes['pictures'].attachables[index]

      # Resize and override original file
      processed = ImageProcessing::Vips.source(file.path)
                                       .resize_to_limit!(1024, 768)
      FileUtils.mv processed, file.path

      # Update the ActiveStorage::Attachment's checksum and other data.
      # Check ActiveStorage::Blob#unfurl
      pictures[index].unfurl processed
    end
  end
end

It works but feels a bit hacky to me, because of the usage of an undocumented Rails method, and the assumption that attachment_changes['pictures'].attachables order is exactly the same as pictures.