User "system:serviceaccount:gitlab:default" cannot get resource "deployments" in API group "apps" in the namespace "gitlab"

Question

I am trying to run a k8s deployment job from gitlab kubernetes executor.

I deployed the kubernetes runner using helm as following.

my values.yaml includes the following rbac rules:

rbac:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["list", "get", "watch", "create", "delete"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get"]
    - apiGroups: [""]
      resources: ["pods/attach"]
      verbs: ["list", "get", "create", "delete", "update"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["list", "get", "create", "delete", "update"]      
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["list", "get", "create", "delete", "update"]
    - apiGroups: [""]
      resources: ["services"]
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
    - apiGroups: ["apps"]
      resources: ["deployments"]
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]


  clusterWideAccess: true

  podSecurityPolicy:
    enabled: false
    resourceNames:
    - gitlab-runner

then

helm install --namespace gitlab gitlab-runner -f values.yaml gitlab/gitlab-runner

and, my .gitlab-ci.yml has the following stage:

  script:
    - mkdir -p /etc/deploy
    - echo $kube_config |base64 -d > $KUBECONFIG
    - sed -i "s/IMAGE_TAG/$CI_PIPELINE_ID/g" deployment.yaml
    - cat deployment.yaml
    - kubectl apply -f deployment.yaml

and, I got the following error in pipeline logs:

$ kubectl apply -f deployment.yaml
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "java-demo", Namespace: "gitlab"
Object: &{map["apiVersion":"apps/v1" "kind":"Deployment" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"java-demo" "namespace":"gitlab"] "spec":map["replicas":'\x01' "selector":map["matchLabels":map["app":"java-demo"]] "template":map["metadata":map["labels":map["app":"java-demo"]] "spec":map["containers":[map["image":"square2019/dummy-repo:555060965" "imagePullPolicy":"Always" "name":"java-demo" "ports":[map["containerPort":'\u1f90']]]]]]]]}
from server for: "deployment.yaml": deployments.apps "java-demo" is forbidden: User "system:serviceaccount:gitlab:default" cannot get resource "deployments" in API group "apps" in the namespace "gitlab"
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=services", GroupVersionKind: "/v1, Kind=Service"
Name: "java-demo", Namespace: "gitlab"
Object: &{map["apiVersion":"v1" "kind":"Service" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"java-demo" "namespace":"gitlab"] "spec":map["ports":[map["name":"java-demo" "port":'P' "targetPort":'\u1f90']] "selector":map["app":"java-demo"] "type":"ClusterIP"]]}
from server for: "deployment.yaml": services "java-demo" is forbidden: User "system:serviceaccount:gitlab:default" cannot get resource "services" in API group "" in the namespace "gitlab"
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

Am I missing some RBAC rules here?

thank you!

=== update 2022.06.04 =====

kubectl get role -n gitlab -o yaml
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""

=== update 2022.06.05 ===

Looking at the logic in https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/templates/role.yaml, I modified values.yaml with

clusterWideAccess: false

and, now i get the role as:

k get role -n gitlab -o yaml
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    annotations:
      meta.helm.sh/release-name: gitlab-runner
      meta.helm.sh/release-namespace: gitlab
    creationTimestamp: "2022-06-05T03:49:57Z"
    labels:
      app: gitlab-runner
      app.kubernetes.io/managed-by: Helm
      chart: gitlab-runner-0.41.0
      heritage: Helm
      release: gitlab-runner
    name: gitlab-runner
    namespace: gitlab
    resourceVersion: "283754"
    uid: 8040b295-c9fc-47cb-8c5c-74cbf6c4d8a7
  rules:
  - apiGroups:
    - ""
    resources:
    - pods
    verbs:
    - list
    - get
    - watch
    - create
    - delete
  - apiGroups:
    - ""
    resources:
    - pods/exec
    verbs:
    - create
  - apiGroups:
    - ""
    resources:
    - pods/log
    verbs:
    - get
  - apiGroups:
    - ""
    resources:
    - pods/attach
    verbs:
    - list
    - get
    - create
    - delete
    - update
  - apiGroups:
    - ""
    resources:
    - secrets
    verbs:
    - list
    - get
    - create
    - delete
    - update
  - apiGroups:
    - ""
    resources:
    - configmaps
    verbs:
    - list
    - get
    - create
    - delete
    - update
  - apiGroups:
    - ""
    resources:
    - services
    verbs:
    - get
    - list
    - watch
    - create
    - update
    - patch
    - delete
  - apiGroups:
    - apps
    resources:
    - deployments
    verbs:
    - get
    - list
    - watch
    - create
    - update
    - patch
    - delete
kind: List
metadata:
  resourceVersion: ""

service account and RoleBinding

k get sa -n gitlab
NAME            SECRETS   AGE
default         1         3d2h
gitlab-runner   1         2d2h

k get RoleBinding -n gitlab
NAME            ROLE                 AGE
gitlab-runner   Role/gitlab-runner   9h

however, the same error persists.

=== update 2022.06.06 ===

I applied the following to fix the issue for the moment

kubectl create rolebinding --namespace=gitlab gitlab-runner-4 --role=gitlab-runner --serviceaccount=gitlab:default

Answer 1

I had the same issue and solve it by defining service_account in config.toml for runner:

runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        service_account = "gitlab-runner"

Hope it helps!