python issue os.system commands within a prescribed folder from external in parallel

22 views Asked by At

It is difficult to find a good title.

I think the problem is widely familiar.

I have several UseCase folders, and in each there is a python script config.py. Each config.py performs correctly iff run from a terminal in that very UseCase folder.

I'd like to launch each config.py automatically from yet another python script purge.py. And of course I want to do it in parallel.

The following is my attempt. It works when launch_config is called in a naive for-loop.

def launch_config(project): 
    os.chdir(project)
    os.system('python config.py')

# build all projects in parallel
pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
#
for (filename,name) in targets:
    #
    project = os.path.join(usecase_dir,f"UseCase_{filename}")
    #
    pool.apply_async(launch_config, args=(project,) )
#
pool.close()
pool.join()
#
os.chdir(pwd)

When run in the pool, however, it just crashes from within python itself with a several-hundred-lines-long unintelligible error message. Are there any general mis-assumptions in my approach that can be identified?

0

There are 0 answers