How to write data to stdin of the first process in a Python shell pipeline?

Question

I see this code snippet referenced quite a lot during discussions around Python subprocess pipelines. Obligatory link: https://docs.python.org/3.4/library/subprocess.html#replacing-shell-pipeline

Modified slightly:

p1 = subprocess.Popen(['cat'],
                      stdin=subprocess.PIPE,
                      stdout=subprocess.PIPE)
p2 = subprocess.Popen(['head', '-n', '1'],
                      stdin=p1.stdout,
                      stdout=subprocess.PIPE)
# Allow p1 to receive a SIGPIPE if p2 exits.
p1.stdout.close()
output = p2.communicate()[0]

This shell pipeline is pointless, except to succinctly demonstrate the challenge. Input "abc\ndef\nghi\n" and only "abc\n" should be captured in output.

What is the best way to write data to p1.stdin? I am aware of the input argument to subprocess.Popen.communicate(), but it won't work in a pipeline. Also, the solution needs to handling blocking correctly.

My guess: Reverse engineer the code behind communicate() and create another version for this specific issue. Before I do that, I want to ask if there is a simpler solution of which I am not aware.

Answer 1

You need to call an equivalent of p1.communicate(b"abc\ndef\nghi\n") and output = p2.communicate()[0] concurrently. A portable way to do it is to use threads or asyncio.

Answer 2

Write to p1.stdin and then close it before calling p2.communicate():

In [1]: import subprocess

In [2]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:p1 = subprocess.Popen(['cat'],
:                      stdin=subprocess.PIPE,
:                      stdout=subprocess.PIPE)
:p2 = subprocess.Popen(['head', '-n', '1'],
:                      stdin=p1.stdout,
:                      stdout=subprocess.PIPE)
:p1.stdout.close()
:--

In [3]: p1.stdin.write(b'This is the first line.\n')
Out[3]: 24

In [4]: p1.stdin.write(b'And here is the second line.\n')
Out[4]: 29

In [5]: p1.stdin.close()

In [6]: p2.communicate()
Out[6]: (b'This is the first line.\n', None)

(Don't forget the newlines in the data you send to cat, or it won't work.)

Answer 3

As a working example:

import subprocess, threading

# Unmodified from original code
p1 = subprocess.Popen(['cat'],
                      stdin=subprocess.PIPE,
                      stdout=subprocess.PIPE)
p2 = subprocess.Popen(['head', '-n', '1'],
                      stdin=p1.stdout,
                      stdout=subprocess.PIPE)

# New code: Start a thread that writes stdin, and closes it when complete
def write_stdin():
    p1.stdin.write("abc\ndef\nghi\n")
    p1.stdin.close()

write_t = threading.Thread(target = write_stdin)
write_t.start()

# Unmodified from original code
p1.stdout.close()
output = p2.communicate()[0]