I've been playing around with IPython.parallel and I wanted to use some custom modules of my own, but haven't been able to do it as explained on the cookbook using dview.sync_imports()
. The only thing that has worked for me was something like
def my_parallel_func(args):
import sys
sys.path.append('/path/to/my/module')
import my_module
#and all the rest
and then in the main just to
if __name__=='__main__':
#set up dview...
dview.map( my_parallel_func, my_args )
The correct way to do this would in my opinion be something like
with dview.sync_imports():
import sys
sys.path.append('/path/to/my/module')
import my_module
but this throws an error saying there is no module named my_module
.
So, what is the right way of doing it using dview.sync_imports()
??
The problem is that you're changing the
PYTHONPATH
just in the local process running the Client, and not in the remote processes running in theipcluster
.You can observe this behaviour if you run the next piece of code:
Basically all the modules that you want to use with
sync_imports
must already be in thePYTHONPATH
.If it's not in the
PYTHONPATH
then you must add it to the path in the function that you execute remotely, and then import the module in the function.