How to use requirements.txt with direct github source in virtual environment

Question

Sometimes I must use python modules from github projects. I can do it in my my_requirements.txt as

git+https://github.com/zackhodari/tts_data_tools

And I use python virtual environment for experiments because I do not want lots of garbage on my machine.

pip install -r my_requirements.txt

This is approach work all right for good quality github projects with good requirements.txt. Unfortunately there lots of other projects with inconsistent requirements.txt. Those projects requires some modules pre-installation for compilation.

Ok. I wrote in my my_requirements.txt something like

numpy
git+https://github.com/zackhodari/tts_data_tools

and get "module numpy not found" error because PIP do compilation of github project not in my virtual environment.

What must I do for github projects compilation inside my environment

Here is fulk error message

Collecting git+https://github.com/zackhodari/tts_data_tools (from -r requirements.txt (line 2))
  Cloning https://github.com/zackhodari/tts_data_tools to c:\users\XXX\appdata\local\temp\pip-req-build-4kv5zmyx
  Running command git clone --filter=blob:none --quiet https://github.com/zackhodari/tts_data_tools 'C:\Users\XXX\AppData\Local\Temp\pip-req-build-4kv5zmyx'
  Resolved https://github.com/zackhodari/tts_data_tools to commit 3c1aff21ab0fbed1bbfd2ba8a5a16d0eb610ffe1
  Preparing metadata (setup.py) ... done
Collecting numpy
  Using cached numpy-1.24.2-cp38-cp38-win_amd64.whl (14.9 MB)
Collecting pyreaper
  Using cached pyreaper-0.0.8.tar.gz (124 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\XXX\AppData\Local\Temp\pip-install-_4t7gi1p\pyreaper_27e0fd80dcb141b98b17dd963414e84f\setup.py", line 8, in <module>
          import numpy as np
      ModuleNotFoundError: No module named 'numpy'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Answer 1

It's a bug in pyreaper: it imports numpy before installing it. So you need to install numpy first.

Unfortunately this cannot be done via my_requirements.txt. If you add numpy in it pip downloads and unpacks numpy but not fully install it so other packages that require numpy cannot import it yet until the entire my_requirements.txt is processed. Installation of any package in my_requirements.txt that requires any other package that isn't yet installed will fail.

Either you fully install numpy before all packages that need it. Or you report the bug, wait until it's fixed and then install any packages without pre-installing numpy.