Why does this pod get a 403 Forbidden when calling the Kubernetes API despite a RoleBinding (same with ClusterRoleBinding)?

Question

I created a pod (an Alpine "BusyBox" to run commands in) which then gets the default service account associated with it. I then created a RoleBinding (and later ClusterRoleBinding when the first didn't work) but it still won't let me call the K8s API.

What am I doing wrong?

First I created a container to run commands in:

# Create a namespace to install our pod
kubectl create namespace one

# Now create a pod that we can run stuff in
kubectl run runner -n one --image alpine -- sleep 3600

Then I created a role binding:

# My understanding of this command is that I'm doing the following:
# 1. Creating a binding for the "default" service account in the "one" namespace
# 2. Tying that to the cluster role for viewing things
# 3. Making this binding work in the "default" namespace, so that it can call
#    the API there FROM its own namespace (one)
kubectl create rolebinding default-view --clusterrole=view --serviceaccount=one:default --namespace=default

Then I connected to the pod's terminal and tried to call the API to list all services in its own namespace:

kubectl exec --stdin --tty use-rest -n one -- /bin/ash

# Now I run all these inside that terminal:

# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# The wget installed with Alpine cannot do SSL
apk --no-cache add ca-certificates
apk add wget

wget --ca-certificate=${CACERT} --header="Authorization: Bearer ${TOKEN}" ${APISERVER}/api/v1/namespaces/$NAMESPACE/services

The above gives the error:

--2021-04-20 01:04:54-- https://kubernetes.default.svc/api/v1/namespaces/default/services/
Resolving kubernetes.default.svc (kubernetes.default.svc)... 10.43.0.1
Connecting to kubernetes.default.svc (kubernetes.default.svc)|10.43.0.1|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-04-20 01:04:54 ERROR 403: Forbidden.

But that should be allowed! I get the same error when using a cluster role binding.

USING:

• k3d version v4.4.1
• k3s version v1.20.5-k3s1 (default)
• Calico

Answer 1

You can only have one ServiceAccount per pod and once you've assigned an account to that pod, the default account no longer applies. I was trying to bind the role to the default account, but passing the token of another account I'd created for the pod.