Why stdin blocks the whole event loop without writing to it?

Question

Recently I'm working for a project which requires monitoring several child processes' stdout simultaneously and write stdin, too. In order to make it, I use pyuv as event loop running on the background thread, and add handles of child process dynamically. Here is the python code emulating the process above:

import pyuv, threading, time
import sys, os
loop = pyuv.Loop()
class Process:
    def __init__(self, loop):
        self.in_pipe  = pyuv.Pipe(loop)
        self.out_pipe = pyuv.Pipe(loop)
        self.err_pipe = pyuv.Pipe(loop)
        self.loop = loop

        pass

    def on_read(self, pipe_handle, data, error):
        print("data --> %s" % data)
        pass

    def write(self, data):
        self.in_pipe.write(data)
        pass

    def on_finish(proc, handle, status, signal):
        print("finish_cb", status, signal)

    def start(self):
        stdin  = pyuv.StdIO(stream=self.in_pipe, flags=pyuv.UV_CREATE_PIPE | pyuv.UV_READABLE_PIPE)
        #stdout = pyuv.StdIO(stream=self.out_pipe, flags=pyuv.UV_CREATE_PIPE | pyuv.UV_WRITABLE_PIPE)
        stdout = pyuv.StdIO(stream=self.out_pipe, flags=pyuv.UV_CREATE_PIPE | pyuv.UV_WRITABLE_PIPE)
        stderr = pyuv.StdIO(stream=self.err_pipe, flags=pyuv.UV_CREATE_PIPE | pyuv.UV_WRITABLE_PIPE)
        s = pyuv.Process.spawn(self.loop, args=["python", "-i"], exit_callback=self.on_finish, stdio=[stdin, stdout,stderr])
        self.out_pipe.start_read(self.on_read)
        self.err_pipe.start_read(self.on_read)

_index = 0

proc_list = []

# add process first
# or the loop will just skip
p = Process(loop)
p.start()
proc_list.append(p)

def _run():
    loop.run()
    print("loop fin")

t = threading.Thread(target=_run)
t.setDaemon(True)
t.start()

while _index < 4:
    _index += 1
    # adding the process handle into the loop
    time.sleep(1)
    p = Process(loop)
    p.start()
    proc_list.append(p)

 #   for i in proc_list:
 #      i.write(b"2+2\n")

What I expected is that after a new Process added, it will print its stdout immediately. Thus I shall see Python's REPL message for 4 times.

However, when I execute it, it sucks, just like this:

(env) ➜  _draft git:(pyuv-watcher) ✗ python libuv.py
data --> b'Python 3.5.2 (default, Jun 28 2016, 08:46:01) \n[GCC 6.1.1 20160602] on linux\nType "help", "copyright", "credits" or "license" for more information.\n'
data --> b'>>> '
[nothing then, until 4 seconds later.]
(env) ➜  _draft git:(pyuv-watcher) ✗

However, when I uncomment last 2 lines,i.e.,writing data into stdin of Processes. it's different:

(env) ➜  _draft git:(pyuv-watcher) ✗ python libuv.py
data --> b'Python 3.5.2 (default, Jun 28 2016, 08:46:01) \n[GCC 6.1.1 20160602] on linux\nType "help", "copyright", "credits" or "license" for more information.\n'
data --> b'>>> '
data --> b'4\n'
data --> b'>>> '
data --> b'Python 3.5.2 (default, Jun 28 2016, 08:46:01) \n[GCC 6.1.1 20160602] on linux\nType "help", "copyright", "credits" or "license" for more information.\n'
data --> b'>>> '
data --> b'4\n'
data --> b'>>> '
data --> b'4\n'
data --> b'>>> '
data --> b'4\n'
data --> b'>>> '
data --> b'Python 3.5.2 (default, Jun 28 2016, 08:46:01) \n[GCC 6.1.1 20160602] on linux\nType "help", "copyright", "credits" or "license" for more information.\n'
data --> b'>>> '
data --> b'>>> '
data --> b'4\n'
data --> b'4\n'
data --> b'>>> '
data --> b'4\n'
data --> b'>>> '
data --> b'4\n'
data --> b'>>> '
data --> b'Python 3.5.2 (default, Jun 28 2016, 08:46:01) \n[GCC 6.1.1 20160602] on linux\nType "help", "copyright", "credits" or "license" for more information.\n'
data --> b'>>> '
data --> b'4\n'
data --> b'>>> '
(env) ➜  _draft git:(pyuv-watcher) ✗ 

It seems that if only add a Process handle without writing data into stdin, the waiting process for stdin will block the whole process. How this happens? And how to interact with different processes simultaneously with one event loop?

Answer 1

You are most likely being hit by undefined behavior. libuv (and by extension, pyuv) is not thread-safe. This means you cannot have a loop running in a background thread and start adding handles to it from another thread.