I have some microservices deployed on Google Kubernetes Engine composed of 3 "e2-medium" nodes with 10GB of disk each.
Every time I perform a new deployment of the services I get the following load shedding event on the nodes:
Stopping container alice
The node was low on resource: ephemeral-storage. Container alice was using 236831, which exceeds its request of 0.
The node disk does not appear to be full and occurs when I deploy instances that are not related to the one that is ejected.
How could ephemeral-storage be increased?
In order to get a sense of your of your ephemeral-storage, you may check the following steps:
Menu>Monitoring>Metrics Explorer>
Resource type: kubernetes node
Metric: Ephemeral Storage
Also have a look at the graph in the following link. Ephemeral storage is backed by boot disk size so increasing boot disk size will give you more ephemeral storage.
If you need to set requests and limits on ephemeral storage, please have a look at this example.