Execute shell command with pipes in Python

Question

I'm new to Python, tried googling, but no help..
I need to call such commands in pipes (Get the oldest pending mail from mailq):

mailq |grep "^[A-F0-9]" |sort -k5n -k6n |head -n 1

The command works in shell.

In Python I wrote the following:

 p = subprocess.Popen( 'mailq |grep \"^[A-F0-9]\" |sort -k5n -k6n |head -n 1', shell=True,
                         stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
 response = p.communicate()[0]

But I get such output:

sort: write failed: standard output: Broken pipe\nsort: write error\n

Wondering what is causing such error?

Answer 1

I think this should work:

p = subprocess.Popen( 'mailq |grep \"^[A-F0-9]\" |sort -k5n -k6n |head -n 1', shell=True,
                         stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)
response = p.stdout.readlines(-1)[0]
print response

prints the first line of the response

Answer 2

Instead of making shell take care of splitting your command to several processes and piping them, do that yourself. See here how to pipe one subprocess stream to another subprocess.

That way you can look up outputs of each step (for example by routing stdout to your stdout, just to debug) and figure out if your whole workflow is OK.

It would look a bit like this:

mail_process = subprocess.Popen('mailq', stdin=PIPE, stdout=PIPE, stderr=STDOUT)
grep_process = subprocess.Popen(['grep', '\"^[A-F0-9]"'], stdin=mail_process.stdout, stdout=PIPE, stderr=STDOUT]
...
head_process = subprocess.Popen(["head", ...], ...)
head_process.communicate()[0]

Answer 3

I'd suggest you use subprocess as written here: http://kendriu.com/how-to-use-pipes-in-python-subprocesspopen-objects

ls = subprocess.Popen('ls /etc'.split(), stdout=subprocess.PIPE)
grep = subprocess.Popen('grep ntp'.split(), stdin=ls.stdout, stdout=subprocess.PIPE)
output = grep.communicate()[0]

This is the pythonic way of using pipes.

Answer 4

Python3

shell = subprocess.run(["./snmp.sh","10.117.11.55","1.3.6.1.4.1.43356.2.1.2.1.1.0"],check=True,capture_output=True)
print(shell)

Shell

#!/bin/bash
args=("$@")

snmpwalk -v 1 -c public ${args[0]} ${args[1]}
output = subprocess.check_output(["awk",'{print$4}'"],input=shell.stdout,capture_output=True)
print(output) 

I was getting an error like this

Output = [Errno 2] No such file or directory: "awk '{print $4}'"

The place where I fixed the error was just adding a pipe at the end of the sh file.

Shell

#!/bin/bash
args=("$@")

snmpwalk -v 1 -c public ${args[0]} ${args[1]} | awk '{print $4}'

Hope it helps somebody