Force gnu-parallel to treat replacement string as command

Question

I want to pass a file containing a list of commands to gnu-parallel while using replacement strings (e.g.: {%}).

Unfortunately, if replacement strings are used, gnu-parallel interprets the commands in the file as arguments to /bin/bash.

Here's what I want to do:

parallel -j 8 'CUDA_VISIBLE_DEVICES=$(({%} - 1)) {}' < commands.txt

where the content of commands.txt is:

/path/to/binary -arg1 a -arg2 1.0
/path/to/binary -arg1 a -arg2 1.1
...
/path/to/binary -arg1 z -arg2 9.9

However, this raises the following error:

/bin/bash: /path/to/binary -arg1 a -arg2 1.0: command not found

I was hoping GNU Parallel to run:

CUDA_VISIBLE_DEVICES=0 /path/to/binary -arg1 a -arg2 1.0

The purpose of the environment variable CUDA_VISIBLE_DEVICES is to make each process run on a different GPU (by default all processes run on the same GPU). If I didn't need CUDA_VISIBLE_DEVICES, the following code would work perfectly:

parallel -j 8 < commands.txt

How can I get around this?

Answer 1

While --colsep may work sometimes, it is not always the correct choice. This will create the files abc and def:

echo 'touch abc\ def' | parallel -v --colsep ' ' A=B {}

Normally it will be better to de-quote the expression using eval:

echo 'touch abc\ def' | parallel -v eval A=B {}

So:

parallel -j 8 'eval CUDA_VISIBLE_DEVICES=$(({%} - 1)) {}' < commands.txt

If you use $(({%} - 1)) a lot, consider making your own replacement string:

echo '--rpl {%-1}\ $_=slot()-1' >> ~/.parallel/config
parallel -j 8 'eval CUDA_VISIBLE_DEVICES={%-1} {}' < commands.txt

Or even:

echo '--rpl '"'"'{CUDA} $_="CUDA_VISIBLE_DEVICES=".(slot()-1)'"'" >> .parallel/config
parallel -j 8 'eval {CUDA} {}' < commands.txt

Answer 2

Using --colsep ' ' does the trick:

parallel -j 8 --colsep ' ' 'CUDA_VISIBLE_DEVICES=$(({%} - 1)) {}' < commands.txt