I'm using subprocess
to run a Unix process and then capture the output. Like so:
command_process = subprocess.Popen(
[command],
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
command_output = command_process.communicate()[0]
log_file = open('command.log', 'a')
log_file.write(command_output)
log_file.write('\n')
log_file.close()
I'm saving the output of the process to a variable, command_output
, which gets dumped into the command.log
file.
- How does python store data in variables?
- Are they in a memory buffer, or are variable values written to disk?
- Is it risky to do this if the output is VERY large?
- What are safer alternatives?
Variables are in a memory buffer. You can see this (on Linux) by running (if you only have one running python process)
Sample output:
You can see it doesn't open any files to write to them. Also, from StackOverflow description of tag
variable
:However, when your computer is running out of memory, it can write contents of the RAM to the disk (i. e. pagefile.sys on Windows, swap partition on Linux). This also usually happens when you send your OS into Hibernate mode.
Writing to swap is a last resort of your OS and is something you should try to avoid, since reading from disk is a LOT slower than reading from RAM (also see Teach yourself programming in 10 years).
So if you're dealing with huge outputs that are big enough to possibly overflow your PC's RAM, you should consider dumping that data to a temporary file on your disk periodically (e. g. every 10MB or so), if possible.
If you want to read from an output stream, I'd also recommend you have a look at this.