Using Python, Cython, and GSL on cluster

Question

I am hoping to run my Python program on a cluster computing system. Since I do not have root access on each node of the cluster, I need to install Python and my modules in a local directory and run my code from there. In order to save time, I am able to do a local Python build on one of the nodes, tar it, and save the tarred Python on a proxy server. Then, when I want to run my code on the rest of nodes I simply wget the tarred Python, untar it, and then run my program. So basically I only need to build Python once, and then can use it repeatedly on the cluster; the code I run on the cluster looks like this:

#!/bin/bash
wget www.proxyserver.Local_Python_Build.tgz
tar xzf Local_Python_Build.tgz
./Local_Python_Build/bin/python my_python_code.py

I am running into a one major issues. First, my main code uses some Cython programs I have written that requires GSL. When I do the initial install of Python, I also do a local build of GSL, as well as compile the Cython code (all of which is included in the tar). I've checked to make sure that everything runs correctly after I build it, but the code does not work once I submit it on the cluster. I keep getting the following error:

ImportError: libgsl.so.0: cannot open shared object file: No such file or directory

Now, I've tried adding the path to the directory where libgsl.so.0 is located in LD_LIBRARY_PATH, but that does not work; ie

export LD_LIBRARY_PATH=path/where/libgsl/is

I am worried that I will need to build GSL each time I run the my code; ie run the following code:

#!/bin/bash
wget www.proxyserver.Local_Python_Build.tgz
tar xzf Local_Python_Build.tgz
install GSL
Compile Cython Code
./Local_Python_Build/bin/python my_python_code.py

This will obviously add to my run time, which I'm hoping to avoid. Anyone ideas? The computing cluster runs Linux.

Thanks!

Answer 1

Building GSL at every run (or Python for that matter) should definitely not be necessary. Assuming, all the nodes have access to some shared file system, you could simply,

1. Build Python, GSL and compile your Cython code in a shared folder
2. In the file that you submit to the batch queue system, you change the environment variable (LD_LIBRARY_PATH, PATH, etc) so it can find the executables and the necessary libraries.

The reason why that could have failed is that path/where/libgsl/is is not mounted on the nodes. Check with the system administrator regarding that issue. The specifics of the batch system used will also influence how to make this work.