I'm currently working with Julia (1.0) to run some parallel code on clusters of an HPC. The HPC is managed with PBS. I'm trying to find a way for broadcasting environment variables over all processes, i.e. a way to broadcast a specific list of environment variables automatically in order to have access to them in every julia worker.

#!/bin/bash
#PBS ...

export TOTO=toto
julia --machine-file=$PBS_NODEFILE my_script.jl

In this example, I will not be able to access to the variable TOTO in each julia worker (via ENV["TOTO"]).

The only way I found to do what I want is to set the variables in my .bashrc but I want this to be script-specific. Another way is to put in my startup.jl file :

@everywhere ENV["TOTO"] = $(ENV["TOTO"])

But it is not script-specific because I have to know in advance which variables I want to send. If I do a loop over ENV keys then I'll broadcast all the variables and then override variables I don't want to.

I tried to use DotEnv.jl but it doesn't work.

Thanks for your time.

1 Answers

0
fredrikekre On Best Solutions

The obvious way is to set the variables first thing in script.jl. You can also put the initialization in a separate file, e.g. environment.jl, and load that on all processes with the -L flag:

julia --machine-file=$PBS_NODEFILE -L environment.jl my_script.jl

where environment.jl would, in this case, contain

ENV["TOTO"] = "toto"

etc.