Question
Why is it that python2.7
when called using a subprocess via python3
does
not have the same sys.path as python2.7
called normally? Specifically,
python2.7
via subprocess does not have the "/path/to/site-packages/"
directory in sys.path.
Context
I'd like to use fabric
to deploy a Django app I'm writing. My problem is that
I've written the app in python3
, but fabric
doesn't have explicit python3
support yet. My workaround, until fabric
is fully compatible with python3
,
is to call the fab
script using subprocess
.
For some reason when I call python2.7
using subprocess
via python3
, I
don't have access to any modules in site-packages
.
python2.7 checks
I've got python2.7
and fabric==1.10.0
installed via Enthought.
$ which python
/Users/.../Library/Enthought/Canopy_32bit/User/bin/python
$ python --version
Python 2.7.6 -- 32-bit
$ which fab
/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab
$ fab --version
Fabric 1.10.0
Paramiko 1.15.1
subprocess checks
I have no problem calling fab
from within python2.7
using subprocess.
$ python
Enthought Canopy Python 2.7.6 | 32-bit | (default, Apr 11 2014, 12:06:39)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('fab --version', shell=True)
'Fabric 1.10.0\nParamiko 1.15.1\n'
I also have no problem calling python2.7
from within python3
using subprocess.
$ python3
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('which python', shell=True)
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/python\n'
>>> subprocess.check_output('python --version', shell=True)
Python 2.7.6 -- 32-bit
b''
DistributionNotFound: Fabric==1.10.0
However, even though my subprocess of python2.7
can "find" the fab script, I
can't call it.
# python3
>>> subprocess.check_output(['which', 'fab'])
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab\n'
>>> subprocess.check_output(['fab', '--version'])
Traceback (most recent call last):
File "/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab", line 5, in <module>
from pkg_resources import load_entry_point
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 2877, in <module>
working_set.require(__requires__)
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 698, in require
needed = self.resolve(parse_requirements(requirements))
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 596, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Fabric==1.10.0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 620, in check_output
raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['fab', '--version']' returned non-zero exit status 1
site-packages not in sys.path
It appears that python2.7
when called using subprocess via python3
does
not have the same sys.path as python2.7
called normally.
As expected, sys.path did not have the Enthought "site-packages"
directory,
which contains the fabric
module.
# python3
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## does not contain '/path/to/Enthought/python2.7/site-packages'
Manually add site-packages to sys.path
To confirm that it's possible: when I manually add the correct
"site-packages"
directory, I can successfully import fabric
.
# python3
>>> subprocess.check_output('python -c\
"import sys; sys.path.append(\'/path/to/Enthought/site-packages\');\
from fabric import version; print version.get_version()"',\
shell = True)
b'1.10.0\n'
Other options?
There's got to be a better way to make sure that python2.7, when
invoked via subprocess from python3, has the same sys.path as python2.7
invoked normally. Can someone more familiar with subprocess weigh in?
Additional thoughts
It's really interesting that python2.7
can spawn another python2.7
via subprocess and that subprocess has the correct site-packages dir
in sys.path.
$ python
>>> import subprocess
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## contains "/path/to/Enthought/python2.7/site-packages"
I also compared the sys.path
's from python3, python3 subprocessed by python3,
and python3 subprocessed by python2.7, and was a bit surprised to find that
all three resulted in the same sys.path
.
subprocess supports an
env
parameter that, if given, will be the environment for the called command -- so make a copy of it, remove any troublesome variables, and pass that copy tosubprocess
: