How to explain the fact that "du -sh /" result on root folder is less than "df -h" result ? And how to fix the gap?

Question

I have few docker containers. I'm facing storage issues.

When I do the following command the host (NOT in a docker container)in order to measure the size of all file

du -sh /

I can see that the total size is 50% of total storage.

And when I do this

df -h 

I can see that I have 20% space free and I used 80%.

I use lsof command in order to find deleted open file and It mentions many file from Docker containers :

lsof -nP | grep '(deleted)'

How to explain the difference ? I miss something with docker configuration ?

Answer 1

The question is a very general one in Unix environments.

In Unixes you can remove a file from the file system but if there is a process which is still holding an open file handle on the file, the file is still stored and kept on the disk, available only via this file handle. As soon as the file handle is dropped (closed or the process terminates), the kernel will take care that the remainder of the file is removed from the disk and the disk space is freed.

Such temp-files will influence the output of df, but they will not appear in the output of du which only scans directories.

This feature often is used for using a temp file which shall automatically be removed at process termination: To get such a thing a process creates a file (by opening it for writing) and keeps the file handle open but removes (unlink(2)) the file itself (i. e. removes the directory entry). Then the process can still write to this file and read from it via the file handle, and it doesn't have to clean up after itself after termination.

Docker stuff often seems to have things like this.

Your solution is to close all these open file handles.

This can be achieved by (from shotgun to scalpel):

1. Rebooting. This is a clear cut and often a good solution.
2. Killing all processes which hold the open file handles. This avoids the reboot and thus can keep all processes unharmed which have nothing to do with the situation.
3. Closing all temp-file descriptors in all processes which hold open file handles. This might be an option, depending greatly on your processes of course.

To see the file handles a process has open, you can probably (depending on your system) have a look at /proc/<PID>/fd/. For each process such a directory exists and represents the file handles.