How to configure kubernetes to allow for long running interactive sessions?

Question

Having a problem here running v1.5.1 using the kubectl run/exec -i --tty feature.

Connections do not get interrupted when data is being sent. So when I launch an interactive container session, with a script that outputs something every now and then, it seems to work.

Based on that, I created a nasty workaround to keep my container sessions going, without my interaction. It is really annoying that these sessions terminate prematurely. They leave garbage behind, that needs to cleaned up in case you are attaching to a running production container.

The workaround (print a space-character to STDERR every 59 seconds as a background process):

# within your container session
$ while :; do sleep 59; echo -n ' ' >&2; done &

See the following output, when I do not launch such a script. It demonstrates that an interactive session terminates prematurely within ~ 2 minutes.

This is really annoying if you are leveraging this functionality to debug running applications and container installations.

$ time kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
Waiting for pod default/busybox to be running, status is Pending, pod ready: false
If you don't see a command prompt, try pressing enter.
/ # Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
Waiting for pod default/busybox to terminate, status is Running
error: timed out waiting for the condition

real    2m4.657s

Re-post from this Github Issue.

Answer 1

I have solved this issue by now. The issue is caused by the Kubernetes API being served over an AWS LoadBalancer (ELB or ALB). Those kill connections after a timeout, leading to a disconnection between kubectl and the Kubernetes API server.

This is not a Kubernetes issue per se, it is related to how you serve the Kubernetes API server to the end user. In this case over a load balancer.

Quick woraround: Increase your timeout or send data over the wire once in a while to keep the connection open.