I have a very simple MPI script using mpi4py
# mpitest.py
from mpi4py import MPI
import time
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
time.sleep(100)
If I run this normally with mpirun everything works fine
$ mpirun --np 4 python mpitest.py # just fine
However if I run this from within Python using the subprocess module then things run, but my interpreter becomes very sluggish
>>> import subprocess
>>> proc = subprocess.Popen(['mpirun', '--np', '2', 'python', 'mpitest.py'])
I've already tried keyword arguments like shell=True
.
Environment
I've installed Python, mpi4py, and mpich using latest Miniconda for Linux
mrocklin@carbon:~/workspace/play$ conda list | grep mpi
mpi4py 2.0.0 py36_2
mpich2 1.4.1p1 0
https://conda.io/miniconda.html
Reproducible Steps
mrocklin@carbon:~/workspace/play$ conda create -n test-mpi python=3.6 mpi4py
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/mrocklin/Software/anaconda/envs/test-mpi:
The following NEW packages will be INSTALLED:
mpi4py: 2.0.0-py36_2
mpich2: 1.4.1p1-0
openssl: 1.0.2l-0
pip: 9.0.1-py36_1
python: 3.6.2-0
readline: 6.2-2
setuptools: 27.2.0-py36_0
sqlite: 3.13.0-0
tk: 8.5.18-0
wheel: 0.29.0-py36_0
xz: 5.2.3-0
zlib: 1.2.11-0
xz-5.2.3-0.tar 100% |################################| Time: 0:00:00 3.79 MB/s
zlib-1.2.11-0. 100% |################################| Time: 0:00:00 5.68 MB/s
#
# To activate this environment, use:
# > source activate test-mpi
#
# To deactivate an active environment, use:
# > source deactivate
#
mrocklin@carbon:~/workspace/play$ source activate test-mpi
(test-mpi) mrocklin@carbon:~/workspace/play$ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> proc = subprocess.Popen(['mpirun', '--np', '2', 'python', 'mpitest.py'])
This can be solved by adding the
stdin=subprocess.DEVNULL
keyword to thesubprocess.Popen
call as follows:It turns out that
mpirun
hijacks the stdin pipe a bit, which then makes it so that many of the keystrokes destined for thepython
process don't arrive.