cannot access the statsd metrics endpoint from withing its pod calling localhost

Question

I am creating a local cluster as below:

# kind-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
    endpoint = ["http://${reg_name}:${reg_port}"]
nodes:
- role: control-plane
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "node=worker_1"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "node=worker_2"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "node=worker_3"

with the command

kind create cluster --name airflow-cluster --config kind-cluster.yaml

then I build a docker image as below:

# Dockerfile
FROM apache/airflow
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         vim \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
RUN apt-get install curl
USER airflow

with the command docker build --pull --tag my-image:0.0.1 . and load it with the command

kind load docker-image my-image:0.0.1 --name airflow-cluster

and then I run airflow as below

helm repo add apache-airflow https://airflow.apache.org
helm repo update
kubectl create ns airflow
helm upgrade --install airflow apache-airflow/airflow --namespace airflow \
    --set images.airflow.repository=my-image \
    --set images.airflow.tag=0.0.1

now If I call the statsd metrics endpoint from within the webserver pod as below:

kubectl exec deploy/airflow-webserver -n airflow -- curl airflow-statsd:9102/metrics

It returns the metrics as expected

But If I call it from within the statsd pod as below

kubectl exec deploy/airflow-statsd -n airflow -- wget localhost:9102/metrics

I get

Connecting to localhost:9102 (127.0.0.1:9102)
wget: can't open 'metrics': Permission denied
command terminated with exit code 1

---

---

Why am I not able to call statsd metrics endpoint from within its pod?

---

Here are some investigations that I have done to understand the root cause of this

kubectl exec deploy/airflow-statsd -n airflow -- whoami

returns nobody

and

kubectl exec deploy/airflow-webserver -c webserver -n airflow -- whoami

returns airflow

if I list the roles kubectl get roles -n airflow it returns

airflow-pod-launcher-role
airflow-pod-log-reader-role

and If describe the airflow-pod-log-reader-rolebinding rolebinding I am going to see that the airflow-webserver service account is listed

and if I check the pods service account the sa of the webserver is airflow-webserver and the sa of the statsd pod is airflow-statsd

However, when I added the service-account airflow-statsd to the airflow-pod-log-reader-rolebinding rolebinding, and I recreated the statsd pod, again the permission issue was present