Is there a way to retrieve kubernetes container's ephemeral-storage usage details?

Question

I create some pods with containers for which I set ephemeral-storage request and limit, like: (here 10GB)

Unfortunately, for some containers, the ephemeral-storage will be fully filled for unknown reasons. I would like to understand which dirs/files are responsible for filling it all, but I did not find a solution to do it.

I tried with df -h, but unfortunately, it will give stats for the whole node and not only for the particular pod/container.

Is there a way to retrieve the kubernetes container's ephemeral-storage usage details?

Answer 1

Pods use ephemeral local storage for scratch space, caching, and for logs. The kubelet can provide scratch space to Pods using local ephemeral storage to mount emptyDir volumes into containers.

Depending on your Kubernetes platform, You may not be able to easily determine where these files are being written, any filesystem can fill up, but rest assured that disk is being consumed somewhere (or worse, memory - depending on the specific configuration of your emptyDir and/or Kubernetes platform).

Refer to this SO link for more details on how by default & allocatable ephemeral-storage in a standard kubernetes environment is sourced from filesystem(mounted to /var/lib/kubelet).

And also refer to kubernetes documentation on how ephemeral storage can be managed & Ephemeral storage consumption management works.

I am assuming you're a GCP user, you can get a sense of your ephemeral-storage usage way: Menu>Monitoring>Metrics Explorer> Resource type: kubernetes node & Metric: Ephemeral Storage

Try the below commands to know kubernetes pod/container's ephemeral-storage usage details :

1. Try du -sh / [run inside a container] : du -sh will give the space consumed by your container files. Which simply returns the amount of disk space the current directory and all those stuff in it are using as a whole, something like: 2.4G.

Also you can check the complete file size using the du -h someDir command.

2. Inspecting container filesystems : You can use /bin/df as a tool to monitor ephemeral storage usage on the volume where ephemeral container data is located, which is /var/lib/kubelet and /var/lib/containers.