multiprocessing.Pool's 'apply_async' in Python not running the target function and no error is displayed

Question

I was using the 'apply_async' function in Python's 'multiprocessing' library, and using a 'Thread Pool' to run certain functions for my program.

The function sent as argument to 'apply_async' did not even execute. I was looking for an error, and did not get any.

Check this sample program below:

    import multiprocessing


    def foo(x):
        print(x)
    

    if __name__ == '__main__':
        multiprocessing.freeze_support()

        with multiprocessing.Pool(10) as pool:
            res = pool.apply_async(foo, args=("Hello",))

It is not printing the expected output, 'Hello', as 'apply_async' is not running the function 'foo' in it.

Answer 1

Use 'res.get()' like below, to get the function to run:

import multiprocessing


def foo(x):
    print(x)


if __name__ == '__main__':
    # Freeze Support's job is interesting, check it out!
    multiprocessing.freeze_support()

    with multiprocessing.Pool(10) as pool:
        res = pool.apply_async(foo, args=("Hello",))
        
        # The Solution
        res.get(timeout=1)

The output:

Hello

Check the official Python Documentation: https://docs.python.org/3/library/multiprocessing.html

Answer 2

It is important to understand exactly why your program isn't working. It has to do with the multiprocessing.Pool context manager, which you start with this line of code:

with multiprocessing.Pool(10) as pool:
    # more code...

When your program exits from this block, the context manager calls the function multiprocessing.Pool.terminate, which is documented as follows:

"Stops the worker processes immediately without completing outstanding work. When the pool object is garbage collected terminate() will be called immediately." https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool

This is not what you want. You want to wait until the workers are done. To do that, place a call to pool.close() and another call to pool.join() inside your with block, like this:

import multiprocessing

def foo(x):
    print(x)

if __name__ == '__main__':
    multiprocessing.freeze_support()

    with multiprocessing.Pool(10) as pool:
        res = pool.apply_async(foo, args=("Hello",))
        pool.close()
        pool.join()

You must call pool.close() before calling pool.terminate(), as the docs indicate.

The worker Processes will start when you call apply_async, but starting up a new Process requires some work from the operating system. That takes a little bit of time. If you call Pool.terminate() before the Processes have time to get going, you will kill the workers before they have a chance to do anything. That's the problem with your script.