Serve static files from archive

Question

Is there a module for apache/nginx to serve static files from archive (zip, tgz, tbz …), so that if there is no file in specified location, then stated archive is asked for that file?

Answer 1

I would suspect that there is not, especially as a fallback when regular files are not found. That said, a CGI script to do it would be fairly simple. The performance loss would most likely be noticeable when under load, however.

Answer 2

I'm not aware of such a module.

If you write your own, I would recommend that you take a look ate the try_files directive http://wiki.nginx.org/HttpCoreModule#try_files and hand the request arguments off to a script, e.g. a php file (see on the wiki page the try_files line ending in: /index.php?q=$uri&$args;).

Performance: That way you do some security checks with php, sort out search engine bots and maybe even memcache some files after you have them unpacked, but that depends on your specific request statistics/patterns.

Some tools or pear packages might allow you to extract files to pipe (stdout) and avoid dumping to the filesystem or unpacking can happen in a ramdisk to speed things up. But again, the way to go depends on your files sizes in order to make something like that reliable.

Answer 3

Another possibility may be using a compressed filesystem, depending on types and distribution of the files also with deduplication.

Pro:

-Has almost the same effect als a .zip file ( Storage wise )

-No change on the Webserver part needed

Cons:

-Possibly new FS for the zip DIR

-Maybe not present under used OS ( e.g. ZFS )

Maybe there's another way if you clarify what you are trying to achieve.

Answer 4

For the case of .tgz and .tbz, most of the performance loss (esp for large archives) should come from the fact you have to read from disk and uncompressing all data up to and including the file you asked for. If you ask for the last file in the archive, then whether a CGI script or a webserver, something will still have to spend the time reading, uncompressing, and discarding all the archive data just to get to your file.

Zip format does allow for random access. If your CGI script is very simple (can be a sh script), and essentially just calls "unzip" with the right argument, then the amount of speedup you could get from having a server module do it would be rather small.

That said, it's kind of crazy if a module for doing this doesn't exist (but no, I haven't been able to find one).

Answer 5

You should have a look at SquashFS, it's a compressed filesystem.

You can think of it as a tar.gz archive, used mainly in LiveCD/DVD/USB ISOs but perfectly applicable to your situation.

Here is a HowTo.

PS: Contrary to other answers you don't need a particular OS to use SquashFS but if you happen to run Solaris or FreeBSD go for ZFS compression, it's just great !