Loop through all files in a directory simultaneously

Question

I have a a loop within a loop The outer loop just goes through each file in a directory and runs the inner loop

Can I select multiple files and have the inner loop running on these files simultaneously

I am using a 32 core 100gb ram server. And I am currently only using 1% CPU going through 1 by 1

I would like to speed this up by running multiple files simultaneously, being able to make the most out of my supercomputer and reduce time

Answer 1

There are several options to parallelise a matlab script.

1. if you have the licence for the Parallel computing toolbox, you can replace the outer loop with a parfor loop. See this.
2. if you do not have that licence, you can use the Multicore third party package. You will need to modify your code to write code for a master and for a slave. See this.
3. if you do not want to re-think your code too much, you can remove the outer loop and accept a filename as argument. Then use GNU parallel to launch as many instances of the script as there are processors in the machines, and keeping doing that until all files are processed. See this.