How to see what a k8s container is writing to ephemeral storage

Question

One of our containers is using ephemeral storage but we don't know why. The app running in the container shouldn't be writing anything to the disk.

We set the storage limit to 20MB but it's still being evicted. We could increase the limit but this seems like a bandaid fix.

We're not sure what or where this container is writing to, and I'm not sure how to check that. When a container is evicted, the only information I can see is that the container exceeded its storage limit.

Is there an efficient way to know what's being written, or is our only option to comb through the code?

Answer 1

Adding details to the topic.

Pods use ephemeral local storage for scratch space, caching, and logs. Pods can be evicted due to other pods filling the local storage, after which new pods are not admitted until sufficient storage has been reclaimed.

The kubelet can provide scratch space to Pods using local ephemeral storage to mount emptyDir volumes into containers.

• For container-level isolation, if a container's writable layer and log usage exceeds its storage limit, the kubelet marks the Pod for eviction.

• For pod-level isolation the kubelet works out an overall Pod storage limit by summing the limits for the containers in that Pod. In this case, if the sum of the local ephemeral storage usage from all containers and also the Pod's emptyDir volumes exceeds the overall Pod storage limit, then the kubelet also marks the Pod for eviction.

To see what files have been written since the pod started, you can run:

find / -mount -newer /proc -print

This will output a list of files modified more recently than '/proc'.

/etc/nginx/conf.d
/etc/nginx/conf.d/default.conf
/run/secrets
/run/secrets/kubernetes.io
/run/secrets/kubernetes.io/serviceaccount
/run/nginx.pid
/var/cache/nginx
/var/cache/nginx/fastcgi_temp
/var/cache/nginx/client_temp
/var/cache/nginx/uwsgi_temp
/var/cache/nginx/proxy_temp
/var/cache/nginx/scgi_temp
/dev

Also, try without the '-mount' option.

To see if any new files are being modified, you can run some variations of the following command in a Pod:

while true; do rm -f a; touch a; sleep 30; echo "monitoring..."; find / -mount -newer a -print; done

and check the file size using the du -h someDir command.

Also, as @gohm'c pointed out in his answer, you can use sidecar/ephemeral debug containers.

Read more about Local ephemeral storage here.

Answer 2

We're not sure what or where this container is writing to, and I'm not sure how to check that.

Try look into the container volumeMounts section that is mounted with emptyDir, then add a sidecar container (eg. busybox) to start a shell session where you can check the path. If your cluster support ephemeral debug container you don't need the sidecar container.