Getting output from `exec.Cmd` in "real-time"

Question

This question is similar to Golang - Copy Exec output to Log except it is concerned with the buffering of output from exec commands.

I have the following test program:

package main

import (
    "fmt"
    "log"
    "os/exec"
)

func main() {
    cmd := exec.Command("python", "inf_loop.py")
    var out outstream
    cmd.Stdout = out
    if err := cmd.Start(); err != nil {
        log.Fatal(err)
    }
    fmt.Println(cmd.Wait())
}

type outstream struct{}

func (out outstream) Write(p []byte) (int, error) {
    fmt.Println(string(p))
    return len(p), nil
}

inf_loop.py, which the above refers to, simply contains:

print "hello"
while True:
    pass

The go program hangs when I run it and doesn't output anything, but if I use os.Stdout instead of out then it outputs "hello" before it hangs. Why is there a discrepancy between the two io.Writers and how can it be fixed?

Some more diagnostic information:

• When the loop is removed from inf_loop.py then "hello" is output from both programs, as expected.
• When using yes as the program instead of the python script and outputting len(p) in outstream.Write then there is output, and the output is usually 16384 or 32768. This indicates to me that this is a buffering issue, as I originally anticipated, but I still don't understand why the outstream structure is being blocked by buffering but os.Stdout isn't. One possibility is that the behaviour is the result of the way that exec passes the io.Writer directly to os.StartProcess if it is an os.File (see source for details), otherwise it creates an os.Pipe() between the process and the io.Writer, and this pipe may be causing the buffering. However, the operation and possible buffering of os.Pipe() is too low-level for me to investigate.

Answer 1

Python buffers stdout by default. Try this program:

import sys
print "hello"
sys.stdout.flush()
while True:
    pass

or run Python with unbuffered stdout and stderr:

cmd := exec.Command("python", "-u", "foo.py")

Note the -u flag.

You see different results when using cmd.Stout = os.Stdout because Python uses line buffering when stdout is a terminal.