Kubernetes | Pod Status - CrashLoopBackOff | Back-off restarting failed container

Question

I am creating mongodb deployment i.e, mongo.yaml and I have also created a secret to take the reference of username and password from mongo-secret.yaml. I am implementing this in windows terminal and in a minikube cluster Running it in default namespace. I have ran below commands respectively:-

PS C:\Users\Safeer_Pasha\kubernetes-sample-project> kubectl apply -f mongo-secret.yaml
secret/mongodb-secret created



PS C:\Users\Safeer_Pasha\kubernetes-sample-project> kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-76pch   kubernetes.io/service-account-token   3      6d18h
mongodb-secret        Opaque                                2      2m11s

Once the secret is created, I have ran deployment:-

PS C:\Users\Safeer_Pasha\kubernetes-sample-project> kubectl apply -f mongo.yaml
deployment.apps/mongodb-deployment created

PS C:\Users\Safeer_Pasha\kubernetes-sample-project> kubectl get deployments
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
mongodb-deployment   0/1     1            0           51s

PS C:\Users\Safeer_Pasha\kubernetes-sample-project> kubectl get pods
NAME                                 READY   STATUS             RESTARTS      AGE
mongodb-deployment-8f6675bc5-hk6zt   0/1     CrashLoopBackOff   3 (43s ago)   95s

PS C:\Users\Safeer_Pasha\kubernetes-sample-project> kubectl describe pod mongodb-deployment-8f6675bc5-hk6zt
Name:         mongodb-deployment-8f6675bc5-hk6zt
Namespace:    default
Priority:     0
Node:         minikube/172.24.73.184
Start Time:   Wed, 20 Apr 2022 15:40:34 +0530
Labels:       app=mongodb
              pod-template-hash=8f6675bc5
Annotations:  <none>
Status:       Running
IP:           172.18.0.4
IPs:
  IP:           172.18.0.4
Controlled By:  ReplicaSet/mongodb-deployment-8f6675bc5
Containers:
  mongodb:
    Container ID:   docker://a9553ef7c260906f5f42f84568d2be919aeba697a98101b8accf24ca73940290
    Image:          mongo
    Image ID:       docker-pullable://mongo@sha256:1e72fdd16fc769e5200dad77eff5b2316730d42473c281d8192872698e1f8689
    Port:           27017/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 20 Apr 2022 15:42:12 +0530
      Finished:     Wed, 20 Apr 2022 15:42:13 +0530
    Ready:          False
    Restart Count:  4
    Environment:
      MONGO_INITDB_ROOT_USERNAME:  <set to the key 'mongo-root-username' in secret 'mongodb-secret'>  Optional: false
      MONGO_INITDB_ROOT_PASSWORD:  <set to the key 'mongo-root-password' in secret 'mongodb-secret'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2p6kf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-2p6kf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m24s                default-scheduler  Successfully assigned default/mongodb-deployment-8f6675bc5-hk6zt to minikube
  Normal   Pulled     2m21s                kubelet            Successfully pulled image "mongo" in 2.758842062s
  Normal   Pulled     2m17s                kubelet            Successfully pulled image "mongo" in 2.699501399s
  Normal   Pulled     2m1s                 kubelet            Successfully pulled image "mongo" in 2.69845367s
  Normal   Created    93s (x4 over 2m21s)  kubelet            Created container mongodb
  Normal   Started    93s (x4 over 2m20s)  kubelet            Started container mongodb
  Normal   Pulled     93s                  kubelet            Successfully pulled image "mongo" in 2.717896979s
  Warning  BackOff    62s (x7 over 2m15s)  kubelet            Back-off restarting failed container
  Normal   Pulling    49s (x5 over 2m23s)  kubelet            Pulling image "mongo"
  Normal   Pulled     46s                  kubelet            Successfully pulled image "mongo" in 2.7922607s

Kindly find files below-

mongo.yaml:-

 apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
        env:
        - name: MONGO_INITDB_ROOT_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-username
        - name: MONGO_INITDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-password

mongo-secret.yaml:-

apiVersion: v1
kind: Secret
metadata:
    name: mongodb-secret
type: Opaque
data:
    mongo-root-username: dXNlcm5hbWU=
    mongo-root-password: cGFzc3dvcmQNCg==

Kindly help me to find the issue and to resolve it. Please provide comments for further details.

After checking the logs (Error occurred) :-

uncaught exception: Error: couldn't add user: Error preflighting normalization: U_STRINGPREP_PROHIBITED_ERROR :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
DB.prototype.createUser@src/mongo/shell/db.js:1367:11
@(shell):1:1
Error saving history file: FileOpenFailed Unable to open() file /home/mongodb/.dbshell: No such file or directory
{"t":{"$date":"2022-04-20T13:44:04.789+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn2","msg":"Connection ended","attr":{"remote":"127.0.0.1:60512","uuid":"453ae58a-1864-478e-95de-6ce536bb77d6","connectionId":2,"connectionCount":0}}

Answer 1

> echo cGFzc3dvcmQNCg==| base64 --decode | cat -e
password^M$

your password in the secret has an extra character at the end. Instead of pre-encoding the secret you can just create it straight:

apiVersion: v1
kind: Secret
metadata:
    name: mongodb-secret
type: Opaque
data:
    mongo-root-username: username
    mongo-root-password: password