ipyparallel with mpi cannot find engines

Question

I'm trying to get a basic ipyparallel environment working using mpi4py as described in the ipyparallel documentation. After starting the ipcluster, I load ipython and try to create a client but it has no IDs and accessing the directview returns a NoEnginesRegistered exception.

Steps I take to get to this point:

1. Create a new environment: conda create --name=ipyparallel and source activate ipyparallel
2. Install ipyparallel and mpi4py: conda install ipyparallel mpi4py
3. Create a new ipython profile: ipython profile create --parallel --profile=mpi
4. Edit ~/.ipython/profile_mpi/ipcluster_config.py and add c.IPClusterEngines.engine_launcher_class = 'MPIEngineSetLauncher'
5. Launch cluster with ipcluster start --profile=mpi

Then I launch ipython and run the following:

import ipyparallel as ipp
c = ipp.client(profile="mpi")
c[:] # <-- NoEnginesRegistered exception

Step 5 reports that "Engines appear to have started successfully" and I can see that a process named "mpiexec" is running. Strangely, I tried these same steps on another machine with the same OS and there it worked with no problems. What am I missing?

Answer 1

I solved the problem for anyone coming here with a similar problem. During installation I had added the notebook extension to jupyter's global config. Not sure why that caused this problem but it's fixed now. Outside of a conda environment, I ran:

sudo pip install ipyparallel
sudo jupyter nbextension disable --py ipyparallel
sudo jupyter nbextension uninstall --py ipyparallel
sudo pip uninstall ipyparallel

and then inside the conda environment I can connect to the ipyparallel engines.