Python PEP 273 and Amazon BotoCore

Question

On a small embedded Linux device with limited space, I am trying to place the large [10 Mb] Amazon (AWS) BotoCore library (https://github.com/boto/botocore) in a zip file to compress it and then import it in my Python Scripts using zipimport as described in PEP273 (https://www.python.org/dev/peps/pep-0273/).

I modified my script to have the following lines at the beginning:

## Use zip imports
import sys
sys.path.insert(0, '/usr/lib/python2.7/site-packages/site-packages.zip') 

The site-packages zip file only has botocore in it and site-packages directory itself has the other modules I use, but excluding botocore, in it.

Here is a listing of that directory:

    /usr/lib/python2.7/site-packages >> ls -rlt
    total 1940
-rw-rw-r-- 1 root root   32984 Jun  8 12:22 six.pyc
-rw-r--r-- 1 root root     119 Jun 11 07:43 README
drwxrwxr-x 2 root root    4096 Jun 11 07:43 requests-2.4.3-py2.7.egg-info
drwxrwxr-x 2 root root    4096 Jun 11 07:43 six-1.9.0-py2.7.egg-info
drwxrwxr-x 2 root root    4096 Jun 11 07:43 python_dateutil-2.4.2-py2.7.egg-info
drwxrwxr-x 2 root root    4096 Jun 11 07:43 jmespath-0.7.0-py2.7.egg-info
-rw-rw-r-- 1 root root    2051 Jun 11 07:44 pygtk.pyc
-rw-rw-r-- 1 root root    1755 Jun 11 07:44 pygtk.pyo
-rw-rw-r-- 1 root root       8 Jun 11 07:44 pygtk.pth
drwxrwxr-x 2 root root    4096 Jun 11 07:44 futures-2.2.0-py2.7.egg-info
drwxrwxr-x 3 root root    4096 Jun 11 07:44 gtk-2.0
drwxrwxr-x 3 root root    4096 Jun 11 07:44 requests
drwxrwxr-x 3 root root    4096 Jun 11 07:44 dbus
drwxrwxr-x 3 root root    4096 Jun 11 07:44 dateutil
drwxrwxr-x 2 root root    4096 Jun 11 07:44 jmespath
drwxrwxr-x 3 root root    4096 Jun 11 07:44 concurrent
drwxrwxr-x 2 root root    4096 Jun 11 07:44 futures
drwxrwxr-x 2 root root    4096 Jun 12 10:42 gobject
drwxrwxr-x 2 root root    4096 Jun 12 10:42 glib
-rwxr-xr-x 1 root root    5800 Jun 12 10:42 _dbus_glib_bindings.so
-rwxr-xr-x 1 root root   77680 Jun 12 10:42 _dbus_bindings.so
-rwxr-xr-x 1 root root 1788623 Jun 12 11:39 site-packages.zip

And here are the contents of that zipfile:

My problem is that I can import boto3 and import botocore just find, but when I try to use some API methods contained therein, I get exceptions like this:

>> Unknown component: enpoint_resolver

or

>> Unable to load data for: aws/_endpoints!

If I remove the zip file after uncompressing it in the site-packages directory and reboot - my script works fine.

How can I leverage zipfile imports to compress this huge library? Thanks!

Answer 1

Unfortunately, this just isn't going to work.

PEP 273 requires library authors to follow certain rules, which this package does not. In particular, it makes use of __file__ rather than pkgutil.get_data() or an equivalent API. As a result, the files must actually exist in the filesystem.

You might try using FUSE to mount the .zip file in the filesystem, so it appears to Python as if it's uncompressed, without actually taking up all that disk space. Just looking through Google, I came up with fuse-zip, which looks like it could be suitable. You'll want to run some benchmarks to ensure it performs well on your system.