Analyze what exactly is being serialized when using pathos multiprocessing

Question

I have a class object that uses pathos.multiprocessing to parallelize one of its more CPU expansive methods. To reduce overhead, I actually moved the method outside of the original class into a reduced class with a smaller data footprint (Whether this is a good design is another question which I posed here).

I tested this first on a Windows system and obtained some performance improvements, although I still assume serialization overhead to be a limiting factor.

However, when I tried the same on a Unix system, the parallelized version runs significantly slower than the sequential one. This surprises me, because although my understanding is limited, I was under the impression that efficient multiprocessing in Python is easier to achieve on a Unix system due to its ability to fork processes.

I would like to investigate what exactly causes this difference in performance for the two systems, but debugging multiprocessing programs has always struck me as a complicated task: To begin with, is there a way to obtain information on what exactly is passed (i.e. serialized and sent) to each subprocess during the execution of ProcessingPool.imap? This information could help me in further improving the design of my code.

Answer 1

I'm the author of dill, multiprocess, and pathos. The easiest thing to do is to use dill.detect.trace which provides a trace of the path used to serialize any object.

For example:

Python 3.8.18 (default, Aug 25 2023, 04:23:37) 
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.detect.trace(True)
>>> class Foo(object):
...   def bar(self):
...     pass
... 
>>> dill.dumps(Foo)
┬ T2: <class '__main__.Foo'>
├┬ F2: <function _create_type at 0x10fc648b0>
│└ # F2 [30 B]
├┬ T1: <class 'type'>
│├┬ F2: <function _load_type at 0x10fc64820>
││└ # F2 [17 B]
│└ # T1 [28 B]
├┬ T1: <class 'object'>
│└ # T1 [15 B]
├┬ D2: <dict object at 0x010f8cb740>
│├┬ F1: <function Foo.bar at 0x10fe16ee0>
││├┬ F2: <function _create_function at 0x10fc64940>
│││└ # F2 [23 B]
││├┬ Co: <code object bar at 0x10fe1b450, file "<stdin>", line 2>
│││├┬ F2: <function _create_code at 0x10fc649d0>
││││└ # F2 [19 B]
│││└ # Co [77 B]
││├┬ D1: <dict object at 0x010f1fdec0>
│││└ # D1 [22 B]
││├┬ D2: <dict object at 0x010fe15d80>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x010fe15c80>
│││├┬ D2: <dict object at 0x010fe15d40>
││││└ # D2 [2 B]
│││└ # D2 [49 B]
││└ # F1 [185 B]
│└ # D2 [230 B]
└ # T2 [349 B]
b'\x80\x04\x95^\x01\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x0c_create_type\x94\x93\x94(h\x00\x8c\n_load_type\x94\x93\x94\x8c\x04type\x94\x85\x94R\x94\x8c\x03Foo\x94h\x04\x8c\x06object\x94\x85\x94R\x94\x85\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x03bar\x94h\x00\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(K\x01K\x00K\x00K\x01K\x01KCC\x04d\x00S\x00\x94N\x85\x94)\x8c\x04self\x94\x85\x94\x8c\x07<stdin>\x94h\x10K\x02C\x02\x00\x01\x94))t\x94R\x94c__builtin__\n__main__\nh\x10NNt\x94R\x94}\x94}\x94(\x8c\x0f__annotations__\x94}\x94\x8c\x0c__qualname__\x94\x8c\x07Foo.bar\x94u\x86\x94b\x8c\x07__doc__\x94Nut\x94R\x94\x8c\x08builtins\x94\x8c\x07setattr\x94\x93\x94h(h#h\x08\x87\x94R0.'

The # denotes an object is completed, and an indentation is that a group of other objects are needed to serialize the current object. Thus, the serialization path is traced out.

This also works for multiprocessing...

>>> import pathos
>>> pathos.multiprocessing.Pool().map(lambda x:x*x, [1,2,3])
F2: <function mapstar at 0x10fe7bca0>
# F2
F1: <function <lambda> at 0x1100ad670>
F2: <function _create_function at 0x10fc64940>
# F2
Co: <code object <lambda> at 0x1100979d0, file "<stdin>", line 1>
F2: <function _create_code at 0x10fc649d0>
# F2
# Co
D3: <dict object at 0x010f1fdec0>
# D3
D2: <dict object at 0x01100b0b40>
# D2
D2: <dict object at 0x01100b0780>
D2: <dict object at 0x01100b0a80>
# D2
# D2
# F1
D2: <dict object at 0x01100b0340>
# D2
F2: <function mapstar at 0x10fe7bca0>
# F2
F1: <function <lambda> at 0x1100ad670>
F2: <function _create_function at 0x10fc64940>
# F2
Co: <code object <lambda> at 0x1100979d0, file "<stdin>", line 1>
F2: <function _create_code at 0x10fc649d0>
# F2
# Co
D3: <dict object at 0x010f1fdec0>
# D3
D2: <dict object at 0x01100b0b40>
# D2
D2: <dict object at 0x01100b0880>
D2: <dict object at 0x01100b0a80>
# D2
# D2
# F1
D2: <dict object at 0x01100b06c0>
# D2
F2: <function mapstar at 0x10fe7bca0>
# F2
F1: <function <lambda> at 0x1100ad670>
F2: <function _create_function at 0x10fc64940>
# F2
Co: <code object <lambda> at 0x1100979d0, file "<stdin>", line 1>
F2: <function _create_code at 0x10fc649d0>
# F2
# Co
D3: <dict object at 0x010f1fdec0>
# D3
D2: <dict object at 0x01100b0b40>
# D2
D2: <dict object at 0x01100b09c0>
D2: <dict object at 0x01100b0a80>
# D2
# D2
# F1
D2: <dict object at 0x01100b0340>
# D2
[1, 4, 9]
>>>