Launch jobs in cache in a loop in bash script

Question

I try to launch multiple experiments, but I want to keep the code script in cache so that any change will not affect experiments waiting to be executed.

The bash script I use is:

for SIZE in 1 2 4 8 16; do
  python main.py $SIZE
done

The problem with my current script is that once I make changes to the code file main.py, the experiments waiting to be executed will use the modified main.py. For example, If python main.py 1 is currently executed, any change to main.py will affect main.py 2, main.py 4... Is there a way to put main.py in cache so that any change to main.py will not affect python main.py $SIZE?

Answer 1

One way to do what you want is to read the code into a Bash variable and use the -c option to run it with Python. Try this Shellcheck-clean Bash code:

#! /bin/bash -p

python_code=$(< main.py)

for size in 1 2 4 8 16; do
    python -c "$python_code" "$size"
done

• See the command substitution section of the Bash Reference Manual for an explanation of $(< main.py).
• This will not work if main.py contains binary data, particularly NUL characters (ASCII 0). That is because Bash variables cannot hold NUL characters.
• I replaced the variable SIZE with size because ALL_UPPERCASE variable names are best avoided since there is a danger of clashes with the large number of special ALL_UPPERCASE variables that are used in shell programming. See Correct Bash and shell script variable capitalization.
• See the When Should You Quote? section of Quotes - Greg's Wiki for an explanation of why I used quotes on both of the variable expansions. (The critical thing to know is that $var without quotes often does not expand to the string contained in the variable var. "$var" always does.)