Kubernetes: Back-off restarting failed container ingress-nginx-controller while trying to update nginx configuration template

Question

So I've been trying to customize the nginx configuration of my ingress-nginx controller that is enabled inside my minikube. I thought about following the steps of Custom NGINX template

that's my deployment.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        gcp-auth-skip-secret: "true"
    spec:
      serviceAccountName: ingress-nginx
      containers:
      - name: ingress-nginx-controller
        image: registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
        volumeMounts:
          - mountPath: /etc/nginx/template
            name: nginx-template-volume
            readOnly: true
      volumes:
        - name: nginx-template-volume
          configMap:
            name: nginx-template
            items:
            - key: nginx.tmpl
              path: nginx.tmpl

in the same folder I got my custom nginx.tmpl file When I apply that deployment the pod never starts with the following errors: ingress-nginx-controller-99cbbd675-6sqlr 0/1 CrashLoopBackOff 2 (4s ago) 20s

When I describe the pod, here's what I get:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  2m19s                 default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-99cbbd675-6sqlr to minikube
  Normal   Pulled     48s (x5 over 2m18s)   kubelet            Container image "registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" already present on machine
  Normal   Created    48s (x5 over 2m18s)   kubelet            Created container ingress-nginx-controller
  Normal   Started    48s (x5 over 2m18s)   kubelet            Started container ingress-nginx-controller
  Warning  BackOff    47s (x10 over 2m16s)  kubelet            Back-off restarting failed container ingress-nginx-controller in pod ingress-nginx-controller-99cbbd675-6sqlr_ingress-nginx(0c542d9b-a052-457c-8d92-225db6df9287)

Full description of deployment:

Name:             ingress-nginx-controller-99cbbd675-6sqlr
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.49.2
Start Time:       Tue, 20 Feb 2024 13:09:05 +0100
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=99cbbd675
Annotations:      <none>
Status:           Running
IP:               10.244.1.234
IPs:
  IP:           10.244.1.234
Controlled By:  ReplicaSet/ingress-nginx-controller-99cbbd675
Containers:
  ingress-nginx-controller:
    Container ID:   docker://1fb60b233cfe467fddd2eaac5453754f886c19d10118cadb0a560c4bc8bccd7a
    Image:          registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Image ID:       docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 20 Feb 2024 13:10:36 +0100
      Finished:     Tue, 20 Feb 2024 13:10:36 +0100
    Ready:          False
    Restart Count:  4
    Environment:    <none>
    Mounts:
      /etc/nginx/template from nginx-template-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6qbmn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  nginx-template-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      nginx-template
    Optional:  false
  kube-api-access-6qbmn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  2m19s                 default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-99cbbd675-6sqlr to minikube
  Normal   Pulled     48s (x5 over 2m18s)   kubelet            Container image "registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" already present on machine
  Normal   Created    48s (x5 over 2m18s)   kubelet            Created container ingress-nginx-controller
  Normal   Started    48s (x5 over 2m18s)   kubelet            Started container ingress-nginx-controller
  Warning  BackOff    47s (x10 over 2m16s)  kubelet            Back-off restarting failed container ingress-nginx-controller in pod ingress-nginx-controller-99cbbd675-6sqlr_ingress-nginx(0c542d9b-a052-457c-8d92-225db6df9287)

Edit: I checked back the pod logs

$ kubectl logs  -n ingress-nginx ingress-nginx-controller-6674b884ff-2bmf9
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.4
  Build:         846d251814a09d8a5d8d28e2e604bfc7749bcb49
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0221 09:13:39.037739       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0221 09:13:39.037821       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0221 09:13:39.041852       7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.3" state="clean" commit="a8a1abc25cad87333840cd7d54be2efaf31a3177" platform="linux/amd64"
I0221 09:13:39.286149       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
F0221 09:13:39.288020       7 main.go:120] Unexpected error obtaining ingress-nginx pod: unable to get POD information (missing POD_NAME or POD_NAMESPACE environment variable

Tried adding POD_NAMESPACE and POD_NAME like this

containers:
      - name: ingress-nginx-controller
        image: registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
          - mountPath: /etc/nginx/template
            name: nginx-template-volume
            readOnly: true

Checked back the logs:

$ kubectl log -n ingress-nginx ingress-nginx-controller-5fd667fc7d-lnvpl 
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.4
  Build:         846d251814a09d8a5d8d28e2e604bfc7749bcb49
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0221 09:30:36.001546       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0221 09:30:36.001625       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0221 09:30:36.005473       7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.3" state="clean" commit="a8a1abc25cad87333840cd7d54be2efaf31a3177" platform="linux/amd64"
I0221 09:30:36.100787       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
E0221 09:30:36.109126       7 store.go:871] unexpected error parsing name and ns: invalid format (namespace/name) found in ''
W0221 09:30:36.109149       7 store.go:875] Unexpected error reading configuration configmap: resource name may not be empty
F0221 09:30:36.110646       7 nginx.go:172] Invalid NGINX configuration template: template: nginx.tmpl:424: function "builds" not defined

This is the configmap is created using this command:

kubectl create configmap nginx-config --from-file=ingress-nginx/nginx.tmpl -n ingress-nginx

nginx.tmpl is the same template provided by the official k8s' image just added some headers inside the http block

Help is really appreciated!