Low on ephemeral Storage

Question

I am trying to deploy a pod and copy 10GB of TPCDS data into the pod. I am using a PVC with storage capacity of 50 GB. My specifications are:

resources:
        limits:
          cpu: "1"
          memory: 20Gi
          #ephemeral-storage: 20Gi
        requests:
          cpu: "1"
          memory: 20Gi

But still I am facing this issue while copying data into the pod.

The node was low on resource: ephemeral-storage. Container spark-kafka-cont was using 10542048Ki, which exceeds its request of 0.

Answer 1

Your issue is related to the present running pods that must be terminated due to insufficient disk space. Updating to a bigger disk helps to delay the eviction process.

Looks like that particular node does not have enough storage available. Suggest you to explicitly specify the local ephemeral storage request and limit (set limits/requests on ephemeral-storage on all your workloads), so that kubernetes will respect that. Otherwise, it is possible that the pod will get evicted. Refer to official docs on Local ephemeral storage & Local Storage Capacity Isolation Reaches GA for details.

To understand and resolve your issue quickly, use kubectl top command to list all the running nodes and pods along with their resource utilization. And also kubectl exec to get a shell in the pod, and then use normal Unix commands like

du -sh [run inside a container] df -h and du -h someDir commands to find where more space is being utilized. ​​​​​​​Sometimes disk space is taken by logs or emptyDir that are causing this issue. Just get an external volume, map it into the container, and get the logs outside of the node. Using the kubectl logs command helps to find out these logs. Check processes you may be configured to log to a file, In that case set your logging setup to log to stdout.