Multiprocessing issue in with python

Question

I am new to multiprocessing in python and have run into some issues with Pool: OS: Mac Monterey M1 chip Python 3.9.12

in module.py: I have tried

def foo(x)
...
return y
pool = mp.Pool(8)
results = pool.map_async(foo, args)

also tried pathos:

def foo(x)
...
return y
pool = ProcessPool(8)
results = pool.amap(foo, args)

in main script:

import module

def main():
   test = module.foo(x)

if __name__ == "__main__":
   main()

I also installed the package to local using setup.py.

current error message including: for multiprocessing

AttributeError: Can't pickle local object 'search.<locals>.foo

and for pathos

TypeError: no default __reduce__ due to non-trivial __cinit__

P.S.: I am new to stackoverflow. I am trying my best to state the problems. I am not sure what information might need to solve the problem. I have been updating the problem every time people asking for new information as soon as I can. So please be polite.

Answer 1

I'm the author of dill, pathos, and multiprocess. It looks like you have an object that won't serialize. Your question doesn't provide enough information for me to give you a solution that I know will work -- but I can give you some things to try.

1. Try a different serialization variant in dill:

    Python 3.7.15 (default, Oct 12 2022, 04:11:53) 
    [Clang 10.0.1 (clang-1001.0.46.4)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dill
    >>> dill.settings['recurse'] = True

The above will change how objects in the global namespace are handled, so if the function you are interested in passing serializes ok, but there's a unserializable object in the global dict... then the change to the above serialization variant may avoid the "bad" object. You could also identify the object that's causing the failure, and delete it from your current namespace. However, if it's needed by the function you are interested in, then you alternately need to refactor the code.

2. Refactor the code, likely to include a __reduce__ method, or similar (e.g. set/get state methods).

See the __reduce__ docs in pickle. The error you are seeing from pathos tells me that you are running in to an object that is written in C (it has a __cinit__) and thus needs some additional methods to tell pickle or dill how to save its state. The problem you might have here is that you didn't create the object in question, and it comes from some module you are importing. This leads to case #3, but without details on your code, I can't say much more.