Any way to use k8s exec liveness probes in a distroless container image?

Question

We have a background processing pod running on AKS which doesn't expose any HTTP (or other) endpoint. Thus we used the following exec liveness probe, which worked as we needed to:

# For the liveness probe we check if the file exists and was modified in the last minute. Original source: https://medium.com/spire-labs/utilizing-kubernetes-liveness-and-readiness-probes-to-automatically-recover-from-failure-2fe0314f2b2e
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - '[ $(find /tmp/healthy -mmin -1 | wc -l) -eq 1 ] || false'

Now we had to migrate to a distroless image for our container. This obviously doesn't contain any shell anymore. So I'm wondering if there is still any possible way to use exec liveness probes or is this basically impossible with distroless containers?

Answer 1

Using Kubernetes exec liveness probes in a distroless container image can be challenging because distroless images are minimal and do not contain a shell or many other tools typically found in a standard Linux distribution.

You might consider modifying your distroless image to include a minimal shell or the specific tools you need for your liveness probe. That approach involves customizing the distroless image, which might slightly increase its size but gives you the flexibility to run shell commands.

That is exactly what GoogleCloudPlatform/cloud-sql-proxy had to do, as illustrated by issue 119: PR1832 added an endpoint for connection testing.

It uses a custom built executable: Instead of relying on shell commands, you can create a small custom executable (written in a language like Go, Rust, etc.) that performs the same check as your shell script. That executable can be included in your distroless image and invoked by the exec liveness probe.

If modifying the image is not preferable, consider alternative liveness probe mechanisms, such as a TCP or HTTP probe. That might require adding some minimal endpoint or service to your application specifically for health checking.

Or: deploy a sidecar container alongside your main container in the same pod. The sidecar container can have a shell or necessary tools and can be responsible for checking the liveness of the main container.