Re-importing Python modules within the function being called by multiprocess?

21 views Asked by At

I wrote a function in Python that multiprocesses the copying of directories. I'm using the module multiprocess instead of multiprocessing because I'm running the code in a Jupyter Notebook. When I ran the code at first, I got the error NameError: name 'shutil' is not defined. It took me a while to figure out, but I had to import shutil a second time within the function main(). Is there a reason for this?

from multiprocess import Pool
import shutil

def main(args):
    # Re-import shutil.
    import shutil
    
    # Unpack arguments.
    src_dir = args["src_dir"]
    dst_dir = args["dst_dir"]
    dir_id = args["dir_id"]

    # Copy source directory to a folder in the destination directory.
    shutil.copytree(src_dir, dst_dir / "folder_{}".format(dir_id))

    return "Copied"

# Source and destination directories.
src_directory = Path(r"~\src")
dst_directory = Path(r"~\dst")

# Create argument list.
args = [
    {
     "src_dir": src_directory, 
     "dst_dir": dst_directory, 
     "dir_id": i
    } for i in range(10)
]

# Multiprocess.
with Pool(processes=batch_size) as pool:
    out = pool.map(main, args)

Here's the original traceback before importing shutil within main().

Traceback (most recent call last):
  File "c:\~\lib\site-packages\multiprocess\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "c:\~\lib\site-packages\multiprocess\pool.py", line 48, in mapstar
    return list(map(*args))
  File "C:\~\AppData\Local\Temp\ipykernel_23252\412796622.py", line 5, in main
NameError: name 'shutil' is not defined

I tried importing shutil in the function main().

0

There are 0 answers