Error Installing numpy in Python Virtual Environment on High-Performance Computing (HPC) cluster running Red Hat Enterprise Linux Server 7

Question

Description:

I'm encountering an issue while trying to install the numpy package within a Python virtual environment on a High-Performance Computing (HPC) cluster running Red Hat Enterprise Linux Server 7. The cluster uses Slurm as the workload manager and Anaconda 3-5.3.1 for managing environments. Below are the steps I've followed and the error I'm facing:

Steps:

Loaded Slurm module:

module load slurm/17.11.12

Loaded Anaconda module:

module load anaconda/3-5.3.1

Created a Python virtual environment named abdou_env:

python3 -m venv abdou_env

Activated the virtual environment:

source abdou_env/bin/activate

While in the virtual environment (abdou_env), I attempted to install the numpy package using pip3 install numpy, but encountered the following error:

Collecting numpy
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fffef3c6d50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/numpy/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fffef3c6d10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/numpy/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fffef3c6790>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/numpy/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fffef3c6150>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/numpy/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fffef3c6050>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/numpy/
  ERROR: Could not find a version that satisfies the requirement numpy (from versions: none)
ERROR: No matching distribution found for numpy

Additional Information:

Python version within the virtual environment: 3.7 Pip version within the virtual environment: 19.2.3 It seems like there's an issue with connecting to the package repository to download and install numpy. I've checked the network connectivity and it seems fine. The cluster's module system and environment configuration might also play a role in this.

Expected Solution:

I'm seeking advice on how to troubleshoot and resolve this issue. Is there something specific I need to configure within the HPC cluster's environment or Anaconda setup to make the package installation work within the virtual environment? Any guidance or suggestions would be greatly appreciated.

Thank you in advance for your assistance!

Answer 1

did you try using the correct version of numpy for your anaconda version ? you can find the version here based on your operating system

Answer 2

I think the problem is that the Anaconda version you are using is 3 years old. Please check if your HPC has new version of anaconda using module avail anaconda.

If you can't find a new version of anaconda, my suggestion is to not use the anaconda at all. Create a dummy conda env with new python.

module load anaconda/3-5.3.1
conda create -n "myenv" python=3.9

Now we will use this dummy conda env. to create a python venv.

source activate
conda activate myenv

Now check the python version if it is indeed python 3.9.

python3 --version

Once you confirm this. Create a python venv.

python3 -m venv --copies abdou_env

Now you need to deactivate the anaconda module. The easiest way is to just logout from the HPC.

Once you login again, you should be able to access the python venv without the anaconda.

source abdou_env/bin/activate

Now you should be able to install packages.

pip install numpy