Parallelize a for loop with pool.imap

Question

I could use some help. I have a process, that is completely independent from one another. Currently it works as a for loop. Since there are quite many repetitions of this same process, it would be much faster being parallelized:

My try so far:

# def jobs_on_rep(n, info, eg, af, n_memory, data, t):
#...
# data.append(new_row)
#...
# return data, map_filename

if __name__ == "__main__":
                        pool = Pool(os.cpu_count())

                    for t in range(0, rep):

                        pool.imap(jobs_on_rep, t)
                        # jobs_on_rep(n, info, eg, af, n_memory, data, t)

In comments the normal for loop. When the entire loop is done, data is exported as csv.

Any advice, why it won't work?

Answer 1

I found a way, that works for me:

import concurrent.futures

rep = 100
i = 0
a = 3
c = 2

def task(i, a, c):
   
    j = i+a+c
    i = (i+a)*2+c
    return i,j

with concurrent.futures.ThreadPoolExecutor() as executor:
    # Submit tasks to the executor
    futures = [executor.submit(task, i, a, c) for i in range(rep)]
    # Collect the results
    results = [future.result() for future in concurrent.futures.as_completed(futures)]
print(results)