Simultaneously using multiple views in the iPython Notebook

Question

I've got a question I'm hoping someone can help me figure out. I'm trying to construct two different parallel views in an iPython notebook. The first view has the processor with ID 0, and the second has all the rest of the processors. I associate a prefix with each of the views so I can easily run different things on the different processors.

I launch a background thread that does a long calculation using the processors in the second view. While that's running in the background, I try to run a command using the first view, but it doesn't work. I get this error: ValueError: '' is not in list.

So I'm wondering if there's a way to do what I'm trying to do here, or if this is unsupported behavior. In short, I'd like to create two different views using different processors. No processors will be shared between the views. Then I'd like to be able to run a background task that uses one view, while simultaneously using the other view for unrelated tasks.

Here's a small example script that results in the error. I'm not sure how to post a notebook directly, so I've just copied and pasted the python script generated from it.

# <codecell>

from IPython import parallel
cli = parallel.Client()

# <codecell>

view1 = cli[0]
view1.block = True
view1.activate("_one")

# <codecell>

view2 = cli[1:]
view2.block = True
view2.activate("_two")

# <codecell>

%px_two import time
def backFunc():
    for i in range(10):
        %px_two time.sleep(5)
        %px_two print "In bg thread"

# <codecell>

from IPython.lib import backgroundjobs as bg
bgJob = bg.BackgroundJobManager()
bgJob.new('backFunc()')

# <codecell>

%px_one import time
def foreFunc():
    for i in range(10):
        %px_one time.sleep(1)
        %px_one print "In fg thread"


# <codecell>

foreFunc()

As soon as foreFunc() is run, it gives the error:

ValueError: '<IDS|MSG>' is not in list

Any thoughts? I'd appreciate any ideas anyone has.

Answer 1

Short Answer

The sockets used by the Client are not threadsafe, so you cannot use them simultaneously in multiple threads. You can use the cluster simultaneously, but you need to create a separate Client for the background task, which will have its own set of sockets:

rc = parallel.Client()
rc2 = parallel.Client()

view1 = rc[0]
view2 = rc2[1:]

And the rest should work as expected.

PS: What's going on

A Client object is mostly an API around a collection of sockets. Each Client has its own set of sockets, and all Views on one Client use the same sockets. When you share those sockets across threads, it's possible for one thread to get part of a message intended for another thread, garbling the message.

Each message is actually a multi-part message zeromq message, sent or received with zmq.Socket.send/recv_multipart, which amounts to:

multipart = []
for i in range(nparts):
    multipart.append(socket.send/recv())

If two threads are doing this at the same time on the same socket, it's possible for messages to get interleaved, so instead of getting two messages:

['a1', 'a2', 'a3'], ['b1', 'b2', 'b3']

we get

['a1', 'a2', 'b1', 'b2', 'b3'], ['a3']

causing the problem you are seeing. The simplest fix is to use different sockets in different threads. An alternate fix is to use locking to ensure that the multi-part messages are received atomically. Separating sockets per-thread allows you to avoid the need for locking, but it does increase the number of sockets you need to use proportionally to the number of concurrent threads.

PPS ...but IPython.parallel is async

I will finish by asking why you are using the Background job at all. You do not need to use threads to accomplish the task you described, because the Client normally doesn't wait for results from the engines. IPython.parallel is async by nature, so you don't need to wait for jobs to finish in order to submit new ones, or do work locally in your interactive session. I generally do not recommend using block=True for anything other than debugging.