Ephemeral containers in Kubernetes

Question

I create a standard nginx pod and I want to run a sidecar container, but it's not working for me in Kubernetes 1.18 - 1.20.

# kubectl alpha debug nginx --image=busybox --target=nginx
Defaulting debug container name to debugger-6wlb5.

# kubectl attach nginx -c debugger-6wlb5
If you don't see a command prompt, try pressing enter.
error: unable to upgrade connection: container debugger-6wlb5 not found in pod nginx_default 

I have gates feature enabled in apiserver, scheduler & controller-manager. Describe on pod doesn't show errors or something:

Mounts:
    /var/run/secrets/kubernetes.io/serviceaccount from default-token-4jhz2 (ro)
Ephemeral Containers:
  debugger-6wlb5:
    Image:        busybox
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
Conditions:
  Type              Status

What do I miss here?

Answer 1

Background

Unfortunately this is still alfa feature and it's not suitable for production. As stated in Ephemeral Containers documentation.

This page provides an overview of ephemeral containers: a special type of container that runs temporarily in an existing Pod to accomplish user-initiated actions such as troubleshooting. You use ephemeral containers to inspect services rather than to build applications.

Warning: Ephemeral containers are in early alpha state and are not suitable for production clusters.

Tests

I have tested this on Kubeadm with Kubernetes 1.19. Feature Gates flag was set in config files like apiserver, scheduler and controller-manager - --feature-gates=EphemeralContainers=true. Also kubelet was modified. As this is alpha it needs specific steps to make it work.

Option 1

This method is described in Debug Running Pods - Debugging with an ephemeral debug container.

$ kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never
pod/ephemeral-demo created
$ kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
Defaulting debug container name to debugger-hnr2w.
If you don't see a command prompt, try pressing enter.
/ # ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    7 root      0:00 sh
   12 root      0:00 ps aux
/ # Now, you are in the debugger-hnr2w container which is in the same pod as container ephemeral-demo
/ # exit
$ 

Describe pod

Containers:
  ephemeral-demo:
    Container ID:   docker://e24cf29efdab9fbf8180ec6c8a9539cdfbdfc490b3b4a4d5dd02d419887c8486
    Image:          k8s.gcr.io/pause:3.1
    ...
Ephemeral Containers:
  debugger-hnr2w:
    Container ID:   docker://649d020e92eedbe03d281fb2368f33ceec176eaaa340fb52af5bf59fc269c701
    Image:          busybox
    ...

In example above, attach was made by using -i flag. It's stated in Copying a Pod while adding a new container.

The -i flag causes kubectl debug to attach to the new container by default. You can prevent this by specifying --attach=false.

Side Note

There second part of quotation is that If your session becomes disconnected you can reattach using kubectl attach. however it applies to another method, not Ephemeral Containers.

Option 2

This option and all prerequisite are described in Ephemeral Containers — the future of Kubernetes workload debugging article. Instead of Deployment I have used nginx pod.

$ kubectl run nginx --image=nginx.

Next you have to create .json file with below configuration. Please change metadata.name to your pod name.

{
    "apiVersion": "v1",
    "kind": "EphemeralContainers",
    "metadata": {
            "name": "nginx"
    },
    "ephemeralContainers": [{
        "command": [
            "sh"
        ],
        "image": "busybox:latest",
        "imagePullPolicy": "IfNotPresent",
        "name": "debugger",
        "stdin": true,
        "tty": true,
        "terminationMessagePolicy": "File"
    }]
}

IMPORTANT

You have to apply it using kubectl replace --raw

$ kubectl replace --raw /api/v1/namespaces/default/pods/<podName>/ephemeralcontainers -f <jsonFileName>.json

Otherwise, you will get error:

kubectl apply -f debug.json 
error: unable to recognize "debug.json": no matches for kind "EphemeralContainers" in version "v1"

After applying proper command:

$ kubectl replace --raw /api/v1/namespaces/default/pods/nginx/ephemeralcontainers -f debug.json
{"kind":"EphemeralContainers","apiVersion":"v1","metadata":{"name":"nginx","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/nginx/ephemeralcontainers","uid":"...}

Use kubectl describe <podname> to verify if EphemeralContainer was created.

$ kubectl describe po | grep 'Container ID' -B 2
Containers:
  nginx:
    Container ID:   docker://a410b326cdc3b95abb2edff8cdb4d7edca9498ba44b54ca6a448967596391813
--
Ephemeral Containers:
  debugger:
    Container ID:  docker://a1357c0daed0ad5664b8c838183a3eb0716339020e829077c14e7438fa5e1cf5

Using this method you will be able to use kubectl attach

$ kubectl attach -it nginx -c debugger
If you don't see a command prompt, try pressing enter.
/ # 

Conclusion

Ephemeral Container is a special type of container that runs temporarily in an existing Pod to accomplish user-initiated actions such as troubleshooting. One you will terminate session, you will not be able to connect again.

You will get error:

$ kubectl attach -it nginx -c debugger
If you don't see a command prompt, try pressing enter.
error: unable to upgrade connection: container debugger not found in pod nginx_default

It might be changed in the future but based on my tests, you can connect to this container only once.