Access kubernetes 'namespaces' object from the kubelet

Question

I need to check the namespace of each pod in my cluster for the presence of a particular label. This needs to be done from the kubelet. I am using the kubernetes go-client to send a REST request to the kube-apiserver and get the namespace object for each pod. Kubelet authenticates and authorizes itself to the kube-apiserver with the user "system:node:". I am maintaining a kubeconfig file for this purpose.

Kubelet user has restricted permissions, rightfully so. It can read only services, endpoints, nodes, pods, PVs, PVCs, secrets etc. (Ref https://kubernetes.io/docs/reference/access-authn-authz/node/). It is understandable that kubelet has only necessary and sufficient permissions for security reasons. I want my kubelet to list/get namespaces. There is a clusterrole and a clusterrolebinding associated with these permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2023-10-18T16:52:25Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node
  resourceVersion: "939101"
  uid: dca5da4a-82b1-4f56-9e20-0f46aa0e0de7
rules:
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - localsubjectaccessreviews
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - create
  - get
  - list
  - watch

I added a block for namespaces.

- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch

But this does not help. I cannot get/list 'namespaces' object as a system:node user. I keep getting below error:

Error from server (Forbidden): namespaces is forbidden: User "system:node:controller-0" cannot list resource "namespaces" in API group "" at the cluster scope

Is 'namespaces' object handled specially by kubernetes in this context? If so, how can I get/list namespace objects using system:node user?

Answer 1

I think his error usually indicates that the user does not have permissions to view namespaces in your cluster. You need to update the IAM policy associated with the node instance profile or configure RBAC roles and bindings for users/groups within your cluster so they can access resources like namespace

Try creating a RoleBinding or ClusterRoleBinding resource as an admin.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ns-creator
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["namespaces"]
  verbs: ["create", "get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: create-ns
subjects:
- kind: User
  name: "system:node:k8s-worker"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: ns-creator
  apiGroup: rbac.authorization.k8s.io

You can also refer to this github link if the issue still exists and also refer to official documentation on namespace walkthrough for more information.