Run a shell script during pip install

Question

I have published a PyPI package which requires some additional Apt packages and so I have a setup.sh in my Github repository which I want to execute along with the installation process of pip install mypackage (i.e, whenever a user runs pip install mypackage it should run the shell script during the process of installation.)

I made my setup.py to use the cmdclass parameter like this

from setuptools import setup, find_packages
from setuptools.command.install import install
import subprocess
import codecs 
import os

class CustomInstall(install):
    def run(self):
        subprocess.check_call("bash setup.sh", shell=True)
        install.run(self)

with open('requirements.txt') as f:
    requirements = f.read().splitlines()


setup(
    ...
    cmdclass={'install': CustomInstall},   install_requires=requirements,
    ...
)

When I normally build it with python setup.py sdist bdist_wheel the Custominstall class runs fine but during pip installation it ignores the Custominstall class.

Finally I found this stackoverflow thread from where I learnt that I only need to run python setup.py sdist which I did and got a tar.gz file in the dist folder but when I go to install that tar.gz file with pip install, I am getting

with open('requirements.txt') as f:

           ^^^^^^^^^^^^^^^^^^^^^^^^

  FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'

What am I doing wrong? Is the path not correct? The requirements file is in my repository's parent folder (not inside any other folder)

Answer 1

The reason why this is happening is because your setup.py depends on requirements.txt but you're not packaging requirements.txt in your distribution. If you open your .tar.gz, you won't find your requirements.txt file. To fix this, create a MANIFEST.in file with the contents:

include requirements.txt

And add include_package_data=True to your setup call.

You'll need to add any other non-python files you want to include in your .tar.gz distribution, such as your setup.sh script as well.

When you build your source distribution with these changes, you'll see that requirements.txt is now included in the contents of your build distribution.

Alternatively, you can also use the package_data keyword in setup for this (or [options.package_data] in setup.cfg). See: this link for more in depth explanation.

---

Generally speaking though, it's not a good idea to run shell scripts or install system packages as part of your setup.py. It will make your package harder to use, not easier.

If your package depends on Python dependencies use install_requires instead.

Answer 2

The immediate problem is that the pip process runs in whichever directory the invoking user happens to be in. Perhaps see also What exactly is current working directory?

The fix for that is How do I get the path and name of the python file that is currently executing? -- basically, examine __file__ and derive the directory you want to access from that.

However, the premise of your question seems deeply flawed. You should declare dependencies on whichever Python packages you depend on, and let pip take it from there; or else, create a Debian package and then declare all your dependencies in terms of other Debian packages. (You can do both, of course. Certainly in the latter case you should probably also do the former, so that your package can be installed on non-Debian platforms.)

You have not told us which packages yours will depend on, so we can only phrase this in the broadest terms. Let's say your package depends on requests and beautifulsoup4. Then your setup.py should declare these dependencies:

setup(
   ...
   install_requires=["requests", "beautifulsoup4"],
   ...
)

For convenience, you can also declare these in requirements.txt, but this file has no formal role in Python packaging. Your dependencies need to be declared in setup.py (or for modern packages, definitely prefer pyproject.toml or setup.cfg).

If some of these dependencies can be fulfilled via the OS package manager, providing another installation mechanism for convenience is strictly optional, but a nice add-on. Your package script should not depend on this (but perhaps vice versa).

#!/bin/sh
apt-get install -y python3-requests python3-bs4

(Perhaps notice also how the Debian packages typically have different names than the corresponding Python packages. This is a nuisance, but basically unavoidable.)