I have deployed csi secret store driver in my cluster and it running as deamon set. When I checked log of deamon set it is showing as below
secrets-store-csi-driver-2dvgp secrets-store E1107 13:15:18.703896 1 reflector.go:140] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
secrets-store-csi-driver-vh9cb secrets-store I1107 13:15:49.105286 1 reflector.go:424] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
secrets-store-csi-driver-vh9cb secrets-store E1107 13:15:49.105349 1 reflector.go:140] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
secrets-store-csi-driver-9zvcr secrets-store I1107 13:15:58.320296 1 reflector.go:424] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
secrets-store-csi-driver-9zvcr secrets-store E1107 13:15:58.320373 1 reflector.go:140] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
secrets-store-csi-driver-ksv9j secrets-store I1107 13:16:07.475520 1 reflector.go:424] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
secrets-store-csi-driver-ksv9j secrets-store E1107 13:16:07.475589 1 reflector.go:140] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
secrets-store-csi-driver-2dvgp secrets-store I1107 13:16:10.201218 1 reflector.go:424] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
secrets-store-csi-driver-2dvgp secrets-store E1107 13:16:10.201278 1 reflector.go:140] "pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
I have below servieaccount,clusterrole and clusterrolebinding setup
kubectl describe ds secrets-store-csi-driver
Name: secrets-store-csi-driver
Selector: app=secrets-store-csi-driver
Node-Selector: kubernetes.io/os=linux
Labels: app=secrets-store-csi-driver
app.kubernetes.io/instance=secrets-store-csi-driver
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=secrets-store-csi-driver
app.kubernetes.io/version=1.3.4
helm.sh/chart=secrets-store-csi-driver-1.3.4
helm.toolkit.fluxcd.io/name=secrets-store-csi-driver
helm.toolkit.fluxcd.io/namespace=kube-system
Annotations: deprecated.daemonset.template.generation: 1
meta.helm.sh/release-name: secrets-store-csi-driver
meta.helm.sh/release-namespace: kube-system
Desired Number of Nodes Scheduled: 8
Current Number of Nodes Scheduled: 8
Number of Nodes Scheduled with Up-to-date Pods: 8
Number of Nodes Scheduled with Available Pods: 8
Number of Nodes Misscheduled: 0
Pods Status: 8 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=secrets-store-csi-driver
app.kubernetes.io/instance=secrets-store-csi-driver
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=secrets-store-csi-driver
app.kubernetes.io/version=1.3.4
helm.sh/chart=secrets-store-csi-driver-1.3.4
Annotations: kubectl.kubernetes.io/default-container: secrets-store
Service Account: secrets-store-csi-driver
Containers:
node-driver-registrar:
Image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=/csi/csi.sock
--kubelet-registration-path=/var/lib/kubelet/plugins/csi-secrets-store/csi.sock
Limits:
cpu: 100m
memory: 100Mi
Requests:
cpu: 10m
memory: 20Mi
Liveness: exec [/csi-node-driver-registrar --kubelet-registration-path=/var/lib/kubelet/plugins/csi-secrets-store/csi.sock --mode=kubelet-registration-probe] delay=30s timeout=15s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/csi from plugin-dir (rw)
/registration from registration-dir (rw)
secrets-store:
Image: registry.k8s.io/csi-secrets-store/driver:v1.3.4
Ports: 9808/TCP, 8095/TCP
Host Ports: 0/TCP, 0/TCP
Args:
--endpoint=$(CSI_ENDPOINT)
--nodeid=$(KUBE_NODE_NAME)
--provider-volume=/var/run/secrets-store-csi-providers
--additional-provider-volume-paths=/etc/kubernetes/secrets-store-csi-providers
--metrics-addr=:8095
--provider-health-check-interval=2m
--max-call-recv-msg-size=4194304
Limits:
cpu: 200m
memory: 200Mi
Requests:
cpu: 50m
memory: 100Mi
Liveness: http-get http://:healthz/healthz delay=30s timeout=10s period=15s #success=1 #failure=5
Environment:
CSI_ENDPOINT: unix:///csi/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from plugin-dir (rw)
/etc/kubernetes/secrets-store-csi-providers from providers-dir-0 (rw)
/var/lib/kubelet/pods from mountpoint-dir (rw)
/var/run/secrets-store-csi-providers from providers-dir (rw)
liveness-probe:
Image: registry.k8s.io/sig-storage/livenessprobe:v2.10.0
Port: <none>
Host Port: <none>
Args:
--csi-address=/csi/csi.sock
--probe-timeout=3s
--http-endpoint=0.0.0.0:9808
-v=2
Limits:
cpu: 100m
memory: 100Mi
Requests:
cpu: 10m
memory: 20Mi
Environment: <none>
Mounts:
/csi from plugin-dir (rw)
Volumes:
mountpoint-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/pods
HostPathType: DirectoryOrCreate
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry/
HostPathType: Directory
plugin-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/csi-secrets-store/
HostPathType: DirectoryOrCreate
providers-dir:
Type: HostPath (bare host directory volume)
Path: /var/run/secrets-store-csi-providers
HostPathType: DirectoryOrCreate
providers-dir-0:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/secrets-store-csi-providers
HostPathType: DirectoryOrCreate
Events: <none>
kubectl describe serviceaccount secrets-store-csi-driver
Name: secrets-store-csi-driver
Namespace: kube-system
Labels: app=secrets-store-csi-driver
app.kubernetes.io/instance=secrets-store-csi-driver
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=secrets-store-csi-driver
app.kubernetes.io/version=1.3.4
helm.sh/chart=secrets-store-csi-driver-1.3.4
helm.toolkit.fluxcd.io/name=secrets-store-csi-driver
helm.toolkit.fluxcd.io/namespace=kube-system
Annotations: meta.helm.sh/release-name: secrets-store-csi-driver
meta.helm.sh/release-namespace: kube-system
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
kubectl describe clusterrole csi-secrets-store-provider-aws-cluster-role
Name: csi-secrets-store-provider-aws-cluster-role
Labels: app=secrets-store-csi-driver-provider-aws
app.kubernetes.io/instance=secrets-provider-aws
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=secrets-store-csi-driver-provider-aws
helm.sh/chart=secrets-store-csi-driver-provider-aws-0.3.4
helm.toolkit.fluxcd.io/name=secrets-provider-aws
helm.toolkit.fluxcd.io/namespace=kube-system
Annotations: meta.helm.sh/release-name: secrets-provider-aws
meta.helm.sh/release-namespace: kube-system
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
serviceaccounts/token [] [] [create]
nodes [] [] [get]
pods [] [] [get]
serviceaccounts [] [] [get]
secret [] [] [list]
ELSBLRM-406344:terraform rajn1$
kubectl describe clusterrolebinding csi-secrets-store-provider-aws-cluster-role
Name: csi-secrets-store-provider-aws-cluster-role
Labels: app=secrets-store-csi-driver-provider-aws
app.kubernetes.io/instance=secrets-provider-aws
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=secrets-store-csi-driver-provider-aws
helm.sh/chart=secrets-store-csi-driver-provider-aws-0.3.4
helm.toolkit.fluxcd.io/name=secrets-provider-aws
helm.toolkit.fluxcd.io/namespace=kube-system
Annotations: meta.helm.sh/release-name: secrets-provider-aws
meta.helm.sh/release-namespace: kube-system
Role:
Kind: ClusterRole
Name: csi-secrets-store-provider-aws-cluster-role
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount secrets-store-csi-driver kube-system
I added list secret permission to the cluster role but deamonset logs is still showing an error