How to resolve python libraries dependencies when using pip

Question

I have come across some nasty dependencies.

Looking around I found solutions like upgrade this, downgrade that...

Some solutions work for some but not for others.

Is there a more 'rounded' way to tackle this issue?

Answer 1

First you need to understand that pip can resolve problems one at a time and when you put it in a corner, it can't go further.

But, if you give to pip the 'big problem' it has a nice way to try to resolve it. It may not always work, but for most cases it will.

The solutions you normally find out there are in some cases a coincidence. Someone has an environment similar to another person and one solution works for both.

But if the solution takes into consideration 'your present environment' then the solution should work for more people than just 'the coincidences'.

Disclaimer: below are linux/terminal commands.

1. Upgrade pip. We want the smartest pip we can get.

pip install --upgrade pip

2. Extract the list of packages you want to install.

In my case (these and many others, trimmed for brevity)

google-cloud-texttospeech attrdict google-cloud-language transformers

3. Give them all at once to pip.

pip install google-cloud-texttospeech attrdict google-cloud-language transformers

It will try all the combinations of versions and dependencies' versions until it finds something suitable. This will potentially download a ton of packages just to see their dependencies, so you only want to make this once.

4. If happy with the result, extract the requirements file.

pip freeze > requirements.txt

This contains all the packages installed, we are not interested in all.

And from it, extract the specific versions of your desired packages.

cat requirements.txt | egrep -i "google-cloud-texttospeech|attrdict|google-cloud-language|transformers"

Note: The command above may not list all your required packages, just the ones that appear in the requirements.txt, so check that all show up.

attrdict==2.0.1
google-cloud-language==1.2.0
google-cloud-texttospeech==2.12.3
transformers==2.11.0

Now you can put that on a file like resolved-dependencies.txt

And next time, install the packages directly with the valid & compatible version with.

pip install -r resolved-dependencies.txt

Answer 2

To test this:

1. I wrote a script to create a virtual environment, attempt to install a requirements file, then delete this environment which I run to quickly test if changes to requirements result in a successful installation or not.

2. I wrote a "test app" which uses the basic functionalities of all the libraries I use in one place, to see if despite a successful installation, there are dependencies that pip is unaware of (due to problematic 3rd party library architecture) that break. I can edit the dependencies of this app, then commit to run build actions that run it and see if it completes or crashes.

This way I can upgrade and add libraries more rapidly.

If all libraries used semantic versioning, declared all their dependencies via requirement files and did not define upper versions in requirement files, this would be a lot simpler. Unfortunately, one of the DB-vendors I use (which I shall not name) is very problematic and has a problematic Python library (amongst many other problems).