I'm trying to reproduce the work of a paper that uses fairseq
code in a Singularity container. In the Singularity recipe I install fairseq
code with pip install --editable .
and it appears to work well:
# singularity build log [...]
Installing collected packages: cython, fairseq
Running setup.py develop for fairseq
Successfully installed cython-3.0.4 fairseq
# [...]
But then I try to execute a fairseq-preprocess
command and I get the following error:
Singularity> fairseq-preprocess --source-lang pt --target-lang pt --trainpref data-bin/sp25000/train --destdir data-bin/sp25000/ --dataset-impl raw
Traceback (most recent call last):
File "/usr/local/bin/fairseq-preprocess", line 6, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3254, in <module>
def _initialize_master_working_set():
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3237, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3266, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 584, in _build_master
ws.require(__requires__)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 787, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'fairseq' distribution was not found and is required by the application
I have done this exact same procedure before in a different computer and it worked, so I'm not sure what I did wrong. I believe the problem may be that the installation is done in an unconventional location. How can I know where it is being installed? There are other issues that can explain this?
If it helps, I'm trying to reproduce this code.
EDIT:
I tried including the path to fairseq in $PYTHONPATH but got the same error.