bash loop in parallel

Question

I am trying to run this script in parallel, for i<=4 in each set. The runspr.py is itself parallel, and thats fine. What I am trying to do is running only 4 i loop in any instance.

In my present code, it will run everything.

#!bin/bash
for i in *
do 
  if [[ -d $i ]]; then
    echo "$i id dir"
    cd $i
      python3 ~/bin/runspr.py SCF &
  cd ..
  else
    echo "$i nont dir"
  fi
done

I have followed https://www.biostars.org/p/63816/ and https://unix.stackexchange.com/questions/35416/four-tasks-in-parallel-how-do-i-do-that but unable to impliment the code in parallel.

Answer 1

You don't need to use for loop. You can use gnu parallel like this with find:

find . -mindepth 1 -maxdepth 1 -type d ! -print0 |
parallel -0 --jobs 4 'cd {}; python3 ~/bin/runspr.py SCF'

Answer 2

Another possible solution is:

find . -mindepth 1 -maxdepth 1 -type d ! -print0 |
xargs -I {} -P 4 sh -c 'cd {}; python3 ~/bin/runspr.py SCF'