Kubernetes : grafana pod behind ingress-nginx

Question

I'm trying to install a grafana pod behind an ingress-nginx on kubernetes. Im on a debian 12 (bookworm) machine. I use the following version of kube :

kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.4

i have install ingress nginx via the command :

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/baremetal/deploy.yaml

i tested my ingress install with the use of

kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo

and that worked fine (http answer : it works)

i have install grafana with the following yaml file :

1-deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ingress-nginx
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:8.1.1
        imagePullPolicy: IfNotPresent
        ports:
        - name: grafana
          containerPort: 3000
          protocol: TCP
        env:
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              name: grafana
              key: admin-user
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana
              key: admin-password
        livenessProbe:
          failureThreshold: 10
          httpGet:
            path: /api/health
            port: grafana
          initialDelaySeconds: 30
          timeoutSeconds: 15
        readinessProbe:
          httpGet:
            path: /api/health
            port: grafana

2-service.yaml

---
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-nginx
  name: grafana
spec:
  type: ClusterIP
  ports:
  - name: grafana
    port: 3000
  selector:
    app: grafana

and a file 0-secret.yaml having my login and password encoded. i used the command

kubectl apply -f grafana

on my directory grafana having those files.

I also have a yaml for my ingress ressource :

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: ingress-nginx
spec:
  ingressClassName: external-nginx
  rules:
  - http:
      paths:
      - path: /login
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 3000

So as far as i understand ingress, it should map my grafana pod throught ingress on my :/login so in my example :

 kubectl get services -A
NAMESPACE       NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default         kubernetes                           ClusterIP   10.96.0.1       <none>        443/TCP                      18h
ingress-nginx   grafana                              ClusterIP   10.104.10.21    <none>        3000/TCP                     43m
ingress-nginx   ingress-nginx-controller             NodePort    10.98.208.172   <none>        80:32445/TCP,443:32416/TCP   18h
ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.106.44.218   <none>        443/TCP                      18h
kube-system     kube-dns                             ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP       18h

:32445/login my browser send me a 503 service temporarily unaivailable error instead of my grafana. I tried to set up my grafana in other namespace, change some ports, expose others, but having each time the same result. Can someone help me to understand what's going on and what's i'm missing? I think it might be a routing issue between kube pods but I don't find any ressources explaining what to setup to solve the problem.

Kind regards, Benjamin

I Installed kubectl kubeadm and kubelet with containerd and calico networking pod I installed ingress for bare metal, and test it with a web service (the test results were fine) I installed grafana on kube and tried to run it behind my ingress with the configurations files describe above. When i go on web browser to try to connect on my grafana via Ingress, i got a 503 http error instead of my grafana login page.