Unable to add TLS certificate to GKE from Google Secret Manager

Question

I have stored the SSL certificates in the GCP secret manager. I'm using Helm to deploy the application and configure the GKE ingress load balancer. I followed this blog to add TLS certificate to GKE from Google Secret Manager.

I stored the certificate in below format in Secret Manager

    -----BEGIN PRIVATE KEY-----  
MIIC2DCCAcCgAwIBAgIBATANBgkqh ...
    -----END PRIVATE KEY-----
    -----BEGIN CERTIFICATE-----  
MIIC2DCCAcCgAwIBAgIBATANBgkqh ...
    -----END CERTIFICATE----

In my helm deployment.yaml file, added the secret as volume

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "test-frontend.fullname" . }}
  namespace: {{ .Values.global.namespace }}
  labels:
    {{- include "test-frontend.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "test-frontend.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "test-frontend.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ include "test-frontend.serviceAccountName" . }}
      automountServiceAccountToken: true
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          envFrom:
          - configMapRef:
              name: {{ include "test-frontend.configMapName" . }}    
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          volumeMounts:
          - name: ispsecret
            mountPath: /var/secret
      volumes:
        - name: testsecret
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "test-tls"    

      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

and created test-secret-provider.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: test-tls
spec:
  provider: gcp
  secretObjects:
  - secretName: test-tls-csi
    type: kubernetes.io/tls
    data: 
    - objectName: "testcert.pem"
      key: tls.key
    - objectName: "testcert.pem"
      key: tls.crt
  parameters:
    secrets: |
      - resourceName: "projects/$PROJECT_ID/secrets/test_ssl_secret/versions/latest"
         fileName: "testcert.pem"

When I deploy the application using Helm, I'm getting below error in the GKE logs

MountVolume.SetUp failed for volume "testsecret" : rpc error: code = InvalidArgument desc = failed to mount secrets store objects for pod test/test-frontend-5f98895b4b-6zq9t, err: rpc error: code = InvalidArgument desc = failed to unmarshal secrets attribute: yaml: line 1: did not find expected key

How to fix this error

Answer 1

Make sure the YAML indentation and format in the parameters section of your SecretProviderClass are correct. YAML is very sensitive to indentation, and even a small mistake can lead to parsing errors.
You would find a similar error in Azure/secrets-store-csi-driver-provider-azure issue 290 for illustration.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: test-tls
spec:
  provider: gcp
  secretObjects:
  - secretName: test-tls-csi
    type: kubernetes.io/tls
    data: 
    - objectName: "testcert.pem"
      key: tls.key
    - objectName: "testcert.pem"
      key: tls.crt
  parameters:
    secrets: |
      - resourceName: "projects/${PROJECT_ID}/secrets/test_ssl_secret/versions/latest"
        fileName: "testcert.pem"

The resourceName/fileName in the parameters section are properly indented as a part of the list under secrets.

And Online YAML Parser seems to error on:

volumeMounts:
            - name: testsecret
            mountPath: /var/secret
        volumes:
        - name: testsecret
        csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
            secretProviderClass: "test-tls" 

The error:

ERROR:

while parsing a block mapping
  in "<unicode string>", line 1, column 5:
        volumeMounts:
        ^
expected <block end>, but found '<block mapping start>'
  in "<unicode string>", line 4, column 11:
              volumes:
              ^

A better indentation:

volumeMounts:
  - name: testsecret
    mountPath: /var/secret
volumes:
  - name: testsecret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "test-tls"

The volumes block was being indented in a way that made it appear as a continuation of the properties of testsecret under volumeMounts, which is not structurally valid.
Now, volumeMounts and volumes are at the same indentation level, indicating they are part of the same Kubernetes resource definition (e.g., a Pod spec).

---

rpc error: code = InvalidArgument 
desc = failed to mount secrets store objects for pod test/test-frontend-5f98895b4b-6zq9t, 
err: rpc error: code = InvalidArgument 
desc = failed to unmarshal secrets attribute: yaml: line 1: did not find expected key 

Check also the logs for the CSI driver pods in your cluster. You can find the CSI driver pods in the kube-system namespace or another namespace if you have configured it differently. Look for any errors that mention issues with processing the SecretProviderClass or accessing the secrets from Google Secret Manager.

Your current error message suggests that there is a failure in parsing the volumeAttributes or similar configuration passed to the CSI driver, specifically indicating a problem with parsing or recognizing a key in the provided YAML configuration.

Check that:

• the volumeAttributes matches the expected keys and structure defined by your CSI driver and the SecretProviderClass. Verify that every key and value under volumeAttributes is expected and supported.

• the secretProviderClass value exactly matches the name of an existing SecretProviderClass in your cluster.

• the configuration within your SecretProviderClasscorrectly references the secret in the Secret Manager. Any discrepancy here, such as an incorrect resourceName or a misconfigured secretObjects section, can lead to errors.

• the format expected by the CSI driver is correct for the CSI convert them into Kubernetes secrets correctly ( a bit as in aws/secrets-store-csi-driver-provider-aws issue 77): review the SecretProviderClass YAML, specifically the parameters.secrets block, to make sure it correctly formats the reference to your secret in Google Secret Manager.

---

In SecretProviderClass Yaml, under parameters.secret block, I'm passing GCP secret manager path projects/$PROJECT_ID/secrets/test_ssl_secret/versions/latest which contains PROJECT_ID.

Will the YAML fetch the PROJECT_ID, or do I need to explicitly mention the PROJECT ID and does it related to permission issue accessing the GCP secret manager?

In the context of Kubernetes and Helm, environment variable placeholders like $PROJECT_ID are not automatically resolved within raw YAML files. For dynamic value substitution (e.g., inserting the PROJECT_ID into your YAML), you typically need to use a templating engine or pass these values in through a process that understands how to replace them. Helm, for example, uses template values from values.yaml and template functions to insert dynamic content into your YAML files before they are applied to the Kubernetes cluster.
See for instance: "How to pull environment variables with Helm charts"

The "secret access permissions" error could indeed be related to a permissions issue if the Kubernetes cluster (specifically, the node pool's service account) does not have permission to access the Google Secret Manager secret. Make sure the service account associated with your GKE nodes has the required roles/permissions (secretmanager.versions.access) to access secrets in Google Secret Manager.

However, the error message you shared suggests that the problem occurs earlier in the process, during the parsing of the YAML configuration, rather than at the point of accessing the secret in Secret Manager.

So make sure that the PROJECT_ID is correctly injected into your SecretProviderClass YAML. Since Helm is being used, you can utilize Helm's templating capabilities to achieve this:

parameters:
  secrets: |
    - resourceName: "projects/{{ .Values.projectID }}/secrets/test_ssl_secret/versions/latest"
      fileName: "testcert.pem"

In the above snippet, {{ .Values.projectID }} is a Helm template directive that tells Helm to replace this placeholder with the value of projectID defined in your Helm chart's values.yaml file. You would need to make sure values.yaml (or whichever values file you are using) includes something like:

projectID: your-gcp-project-id

The revised deployment YAML you have shared looks properly formatted regarding indentation and structure. The persistent error, "failed to unmarshal secrets attribute," points toward an issue with how the secrets data is formatted or interpreted rather than a straightforward syntax error in your YAML.
Considering the context of dynamic variable replacement discussed earlier, the error might be from an improperly formatted or unresolved resourceName in your SecretProviderClass configuration, if the dynamic substitution for PROJECT_ID was not effectively applied.