kubernetes pod (mssql-tools) failing with CrashLoopBackOff error and restarting

Question

I'm using Rancher Dekstop for K8 in WSL 2 in Windows 11. I'm trying to create a pod using the simple yaml:

apiVersion: v1
kind: Pod
metadata:
  name: mssql-tools
  labels:
    name: mssql-tools
spec:
  containers:
  - name: mssql-tools
    image: mcr.microsoft.com/mssql-tools:latest

But it is continuously giving CrashLoopBackOff error.

$ kubectl get pods
NAME          READY   STATUS             RESTARTS      AGE
mssql-tools   0/1     CrashLoopBackOff   11 (8s ago)   14m

And here is the result of kubectl describe pod mssql-tool:

$ kubectl describe pod mssql-tools
Name:             mssql-tools
Namespace:        default
Priority:         0
Service Account:  default
Node:             desktop-2ohsprk/172.22.97.204
Start Time:       Mon, 26 Dec 2022 04:34:19 +0500
Labels:           name=mssql-tools
Annotations:      <none>
Status:           Running
IP:               10.42.0.57
IPs:
  IP:  10.42.0.57
Containers:
  mssql-tools:
    Container ID:   docker://76343010f4344a5d26fb35f3b0278271d3336e8e10d695cc22e78520262f34bf
    Image:          mcr.microsoft.com/mssql-tools:latest
    Image ID:       docker-pullable://mcr.microsoft.com/mssql-tools@sha256:62556500522072535cb3df2bb5965333dded9be47000473e9e0f84118e248642
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 26 Dec 2022 04:46:20 +0500
      Finished:     Mon, 26 Dec 2022 04:46:20 +0500
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 26 Dec 2022 04:45:51 +0500
      Finished:     Mon, 26 Dec 2022 04:45:51 +0500
    Ready:          False
    Restart Count:  9
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wkqlg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-wkqlg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       12m                   default-scheduler  Successfully assigned default/mssql-tools to desktop-2ohsprk
  Normal   Pulled          12m                   kubelet            Successfully pulled image "mcr.microsoft.com/mssql-tools:latest" in 1.459473213s
  Normal   Pulled          12m                   kubelet            Successfully pulled image "mcr.microsoft.com/mssql-tools:latest" in 823.403008ms
  Normal   Pulled          11m                   kubelet            Successfully pulled image "mcr.microsoft.com/mssql-tools:latest" in 835.697509ms
  Normal   Pulled          11m                   kubelet            Successfully pulled image "mcr.microsoft.com/mssql-tools:latest" in 873.802598ms
  Normal   Created         11m (x4 over 12m)     kubelet            Created container mssql-tools
  Normal   Started         11m (x4 over 12m)     kubelet            Started container mssql-tools
  Normal   Pulling         10m (x5 over 12m)     kubelet            Pulling image "mcr.microsoft.com/mssql-tools:latest"
  Normal   Pulled          10m                   kubelet            Successfully pulled image "mcr.microsoft.com/mssql-tools:latest" in 740.64559ms
  Warning  BackOff         6m56s (x25 over 11m)  kubelet            Back-off restarting failed container
  Normal   SandboxChanged  50s                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          48s                   kubelet            Successfully pulled image "mcr.microsoft.com/mssql-tools:latest" in 951.332457ms
  Normal   Pulled          32s                   kubelet            Successfully pulled image "mcr.microsoft.com/mssql-tools:latest" in 828.839917ms
  Normal   Pulling         4s (x3 over 49s)      kubelet            Pulling image "mcr.microsoft.com/mssql-tools:latest"
  Normal   Pulled          3s                    kubelet            Successfully pulled image "mcr.microsoft.com/mssql-tools:latest" in 713.951656ms
  Normal   Created         3s (x3 over 48s)      kubelet            Created container mssql-tools
  Normal   Started         3s (x3 over 48s)      kubelet            Started container mssql-tools
  Warning  BackOff         2s (x5 over 47s)      kubelet            Back-off restarting failed container

The same container works perfectly if I run it via docker and I can use its shell to execute sqlcmd properly.

I can't figure out any reason for this.

Any help would be really appreciated. Thanks

Answer 1

So after trying and digging through multiple options, finally it worked by executing the command sleep 3600000 i.e. delaying it so that the pod initializes itself properly and then executes the container. Here is the working yaml:

apiVersion: v1
kind: Pod
metadata:
  name: mssql-tools
  labels:
    name: mssql-tools
spec:
  containers:    
  - name: mssql-tools
    image: mcr.microsoft.com/mssql-tools:latest
    command: ["sleep"]
    args:
      - "3600000"
    imagePullPolicy: IfNotPresent

The command and argument passing portion can also be mentioned like the following:

apiVersion: v1
...
...
spec:
  containers:    
  - name: mssql-tools
    image: mcr.microsoft.com/mssql-tools:latest
    command:
      - sleep
      - "3600000"
...

and btw, you can also deploy a container by passing a command with the kubectl run command line: i.e.

kubectl run mssql --image=mcr.microsoft.com/mssql-tools --command sleep 3600000 -n myNameSpace

Note: You can omit -n myNameSpace if you are not deploying it in a specific namespace or deploying it in the default namespace.

Answer 2

Crashloopbackoff is the common error which indicates that pod failed to start and it continued to fail repeatedly when kubernetes tried to restart this.

To troubleshoot this issue follow the below steps:

1. Check for “Back off Restarting Failed Container” by running the command Run kubectl describe pod [name].

   If you get a Liveness probe failed and Back-off restarting failed container messages from the kubelet, this indicates the container is not responding and is in the process of restarting.

2. Check from the previous container instance. Run kubectl get pods to identify the Kubernetes pod that causes CrashLoopBackOff error. You can run kubectl logs --previous --tail 10command to get the last ten log lines from the pod.

3. Check deployment logs by running the command: kubectl logs -f deploy/ -n

Refer to this link for more detailed troubleshooting steps.