I have a Python code that does some computation and writes the data in C.csv file. The computation is huge and takes time.
So while my program is running, I want to check if data is being written or not using:
$ less +F C.csv
What I notice is that while my program is running, I do not see any output being written in C.csv but as soon as I give CTRL+C signal, all of a sudden a lot of entries appear in C.csv file.
Now, I know that disk I/O are generally buffered and probably the program will wait for the buffer to get full before it actually writes it to the file (this is my assumption). So, I googled how to check the size of Buffer which suggested me the following method:
import io
print (io.DEFAULT_BUFFER_SIZE)
This returns 8192 (bytes) in my machine. I thought writes will only happen when the data to be written does not fit into buffer, i.e., when the data size crosses 8192 bytes. But when I check the size of C.csv after CTRL+C, it shows 236540 bytes.
How did so much data fit into the buffer? Or is there something else happenning?