Output file contains nothing before script finishing

Question

I write a python script in which there are several print statement. The printed information can help me to monitor the progress of the script. But when I qsub the bash script, which contains python my_script &> output, onto computing nodes, the output file contains nothing even when the script is running and printing something. The output file will contains the output when the script is done. So how can I get the output in real time through the output file when the script is running.

Answer 1

Actually write to the file rather than piping and flush after each write or after each write call sys.stdout.flush() but you are better off using a logger function and replacing the prints with logs.

From Comments: A logger function is one that you call instead of print that will output to somewhere the text, possibly timestamped and with other information, they usually let you output various amounts of information to various destinations including stdout and files. See python 2 or 3 documents for information on pythons built in logging function.

Answer 2

I like to write data to sys.stderr sometimes for this sort of thing. It obviates the need to flush so much. But if you're generating output for piping sometimes, you remain better off with sys.stdout.