Python: contextlib.redirect_stdout sometimes doesn't work in real time

Question

I am using Python3.5 on Ubuntu. The following script creates a file out and fills it with more and more lines of "Hello":

import contextlib
with contextlib.redirect_stdout(open('out','w')):
    while True:
        print('Hello')

However, if I make a tiny modification by adding a call to time.sleep(), the file out stays empty:

import contextlib
import time
with contextlib.redirect_stdout(open('out','w')):
    while True:
        print('Hello')
        time.sleep(1)

If I further change the code by turning the loop into a finite loop, it fills out but only in a lump at the end of the loop.

Can anyone reproduce this and explain it?

Answer 1

This is caused by output buffering. This is an interesting case given the use of contextlib, but the underlying issue is with the open('out', 'w') statement.

To prevent output buffering, you can set the buffering argument to 0 if you're using Python 2:

import contextlib
import time

with contextlib.redirect_stdout(open('out','w', 0)):
    while True:
        print 'Hello'
        time.sleep(1)

Alternatively, the file contents will be written once the context manager closes the file (or once the buffered content exceeds a certain size).

If you're using Python 3, then you can't have unbuffered text I/O (kindly pointed out in the comments). However, flushing stdout at the end of each loop should have the desired effect:

import contextlib
import time
import sys

with contextlib.redirect_stdout(open('out','w')):
    while True:
        print('Hello')
        sys.stdout.flush()
        time.sleep(1)

Answer 2

I bet this has been asked before but I can't find a good duplicate.

This is caused by output buffering. When stdout goes to a terminal, it is line buffered, meaning output is sent from your program to the terminal every time a "\n" is encountered. But when stdout refers to a file, it is block buffered. This means actual output is generated only when the buffer grows full (or when you explicitly flush it or when the handle is closed).