Rails/Dragonfly/Apache - Rack::Cache - how to use X-Sendfile?

Question

I'm using Dragonfly to serve processed images for my Rails app. Dragonfly relies on Rack::Cache for future visits to those processed images, so that Dragonfly won't have to process those images again and again, thus wasting CPU time.

My problem starts here: if I'm right that sending a file via Rack::Cache still busies a Rails process, then viewing a page of 30 images, even if these images have a small file size, will tie up the Rails processes pretty quickly. If a couple more visitors come to see that page, then they will experience very slow response times. How do I get these files served via X-Sendfile?

I've set the following in production.rb, but I know these are for the assets from Rails, not the Dragonfly files:

config.serve_static_assets = false
config.action_dispatch.x_sendfile_header = "X-Sendfile"

I know that Rack::Cache somehow supports X-Sendfile (probably through Rack::Sendfile) because it produces a body that responds to #to_path. However, I don't know how to enable this. When I check files that come from Rack::Cache, I don't see any X-Sendfile information:

Date: Wed, 02 Nov 2011 11:38:28 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.9
Content-Disposition: filename="2.JPG"
Cache-Control: public, max-age=31536000
Etag: "3174d486e4df2e78a5ff9174cacbede5787d4660"
X-Content-Digest: c174408eda6e689998a40db0aef4cdd2aedb3b6c
Age: 28315
X-Rack-Cache: fresh
Content-Length: 22377
Status: 200
Content-Type: image/jpeg

I know, based on posts around the net, that I'm supposed to see something like:

X-Sendfile: /path/to/file

In the end I don't know if its Dragonfly or Rack::Cache (or both) that I have to configure. How do I get either Dragonfly and/or Rack::Cache to serve files via X-Sendfile?

Info about my setup:

• Rails 3.1.1
• Passenger 3.0.9
• CentOS
• Sendfile module is installed, as far as I know. I have XSendFile On and XSendFilePath /path/to/app specified in my virtualhost configuration, and Apache doesn't complain about the directive XSendFile not existing.

Thanks!

UPDATE Nov 6, 2011

Based on this old update, as long as Rack::Sendfile is placed in front of Rack::Cache, then X-Sendfile will be used. I did that, and this is how my middleware looks like. The files, however, still don't have the X-Sendfile tag. Again, I don't know if that is a sure-fire way of determining if X-Sendfile is enabled, so I checked the Passenger queue. It seems that the queue is greatly encumbered when I visit a page.

UPDATE Nov 7, 2011

It seems this is purely a Rack::Cache and Rails 3.1 issue. While Rack::Cache supports the use of X-Sendfile through Rack::Sendfile (like I mentioned above, Rack::Cache, when using the Disk EntityStore since that responds_to to_path since the body it returns is a subclass of File), Rails 3.1 uses its own storage solution. Rails 3.1 uses ActiveSupport::Cache::FileStore, which is set by default, if you don't specify anything in your production.rb file.

The problem with FileStore is that the body it returns, to be part of the response to be sent upstream, because that body doesn't respond to to_path. The body is an instance of ActiveSupport::Cache::Entry. You can see here that when the FileStore is asked to read a cached file, it reads it via File.open('/path/to/file') {|f| Marshal.load(f) } which returns an instance of Entry. The value that ultimately gets passed upstream and back to the client, is Entry#value.

My questions

To help me decide whether I should patch this, or to get Rails to use Rack::Cache's own Disk store instead, I have some questions:

1. What's the reason Rack::Cache's own storage solutions weren't used for Rails 3.1? Why does Rails have its own?
2. Is there a reason Marshal is used? Is there a reason that a bytestream of data should be sent back instead?

I got in deeper than I usually go, and will be surprised if I understood things correctly. I hope to find an answer!

Answer 1

As an alternative to Varnish, you can use Apache's mod_disk_cache. It would be less work to set up as you are already running Apache.

Answer 2

I ended up getting this to work, albeit with nginx & unicorn rather than Apache & Passenger.

As you pointed out in your Github issue, you can switch Rack::Cache back to use it's standard file:/ store rather than the rails:/ store, which will allow the responses to respond to to_path.

config.action_dispatch.rack_cache = {
  :verbose     => true,
  :metastore   => URI.encode("file:/PATH/TO/CACHE/STORE"),
  :entitystore => URI.encode("file:/PATH/TO/CACHE/STORE")
}

Dragonfly does this in development, and you can still do it in production if you like. The caveat with doing this is if you use any of the Rails caching features that use Rack::Cache, the cache entries will be stored in that store rather than the standard Rails one so you'll need to account for that if you need to clear any of those entries manually.

You then also need to make sure that you insert the Rack::Sendfile middleware at the front of the stack with the config.action_dispatch.x_sendfile_header argument. Without the config argument, Rack::Sendfile wont add the header.

config.middleware.insert 0, Rack::Sendfile, config.action_dispatch.x_sendfile_header

My Gist shows my relevant lines in production.rb and my nginx template. Should be easily adapted to work with the Apache X-Sendfile module.

One other thing to note if you're testing this, is that if you only send a HEAD request via cURL for example, you will not get the relevant X-Sendfile header in the response as Rack::Cache wont actually send the body for a HEAD request and so Rack::Sendfile has nothing to call to_path on.