What am I missing in this kubernetes RBAC setup?

Question

I want to run a pod that listens for updates to endpoint lists (I'm not yet ready to adopt the alpha-level feature of endpoint sets, but I'll expand to that eventually.)

I have this code:

package main

import (
    "fmt"
    "os"
    "os/signal"
    "sync"
    "syscall"

    "k8s.io/apimachinery/pkg/util/runtime"
    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/cache"
)

func ReadKubeConfig() (*rest.Config, *kubernetes.Clientset, error) {
    config, err := rest.InClusterConfig()
    if err != nil {
            return nil, nil, err
    }
    clients, err := kubernetes.NewForConfig(config)
    if err != nil {
            return nil, nil, err
    }
    return config, clients, nil
}

func main() {
    _, cs, err := ReadKubeConfig()
    if err != nil {
            fmt.Printf("could not create Clientset: %s\n", err)
            os.Exit(1)
    }
    factory := informers.NewSharedInformerFactory(cs, 0)
    ifmr := factory.Core().V1().Endpoints().Informer()
    stop := make(chan struct{})
    ifmr.AddEventHandler(cache.ResourceEventHandlerFuncs{
            AddFunc: func(next interface{}) {
                    fmt.Printf("AddFunc(%v)\n", next)
            },
            UpdateFunc: func(prev, next interface{}) {
                    fmt.Printf("UpdateFunc(%v, %v)\n", prev, next)
            },
            DeleteFunc: func(prev interface{}) {
                    fmt.Printf("DeleteFunc(%v)\n", prev)
            },
    })
    wg := &sync.WaitGroup{}
    wg.Add(1)
    go func() {
            defer runtime.HandleCrash()
            ifmr.Run(stop)
            wg.Done()
    }()
    ch := make(chan os.Signal, 1)
    signal.Notify(ch, os.Interrupt)
    signal.Notify(ch, os.Signal(syscall.SIGTERM))
    signal.Notify(ch, os.Signal(syscall.SIGHUP))
    sig := <-ch
    fmt.Printf("Received signal %s\n", sig)
    close(stop)
    wg.Wait()
}

I get this error when deploying and running:

kubeendpointwatcher.go:55: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:eng:default" cannot list resource "endpoints" in API group "" at the cluster scope

I have the following role and role binding defined and deployed to the "eng" namespace:

watch_endpoints$ kubectl -n eng get role mesh-endpoint-read -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: "2021-07-08T19:59:20Z"
  name: mesh-endpoint-read
  namespace: eng
  resourceVersion: "182975428"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/eng/roles/mesh-endpoint-read
  uid: fcadcc2a-19d0-4d6e-bee1-78413f51b91b
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get
  - list
  - watch

I have the following rolebinding:

watch_endpoints$ kubectl -n eng get rolebinding mesh-endpoint-read -o yaml | sed -e 's/^/    /g'

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: "2021-07-08T19:59:20Z"
  name: mesh-endpoint-read
  namespace: eng
  resourceVersion: "182977845"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/eng/rolebindings/mesh-endpoint-read
  uid: 705a3e50-2a73-47ed-aa62-0ea48f3493ee
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: mesh-endpoint-read
subjects:
- kind: ServiceAccount
  name: default
  namespace: default

You will note that I apply it to both the default namespace and the eng namespace serviceaccount named default although the error message seems to indicate that it is indeed running in the default serviceaccount in the eng namespace.

I have previously used Role and RoleBinding and ServiceAccount objects that work as expected, so I don't understand why this doesn't work. What am I missing?

For testing/reproduction purposes, I run this program by doing kubectl cp of a built binary (cgo off) into a container created with kubectl -n eng create deplpoy with a vanilla ubuntu image running /bin/sh -c sleep 999999999, and then executing a /bin/bash shell in that pod-container.

Answer 1

You have created role and rolebinding for eng namespace. However, as per the error message:

kubeendpointwatcher.go:55: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:eng:default" cannot list resource "endpoints" in API group "" at the cluster scope

you are doing query for endpoints at the "cluster" scope. Either try to limit your query to eng namespace or use clusterrole/clusterbindings

Error message provide hint(system:serviceaccount:eng:default)that serviceaccount running in eng namespace, whose name is default does not have permission to query ep at cluster scope.

To validate this, you can run two curl calls, first exec into the pod using the same sa and then run the following for eng namespace and later try it on other namespaces.

curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/default/pods