Using worker identity with gcloud

Question

I have 2 docker imags with gcloud sdk and my entrypoint script performs some checks using gcloud, like following

gcloud pubsub subscriptions describe $GCP_SUB_NAME --quiet
result="$?"

if [ "$result" -ne 0 ]; then
    echo "Subscription not found, exited with non-zero status $result"
    exit $result
fi

I am running these in gke...

I have a different GCP Service Account for each docker image which is connected to GKE Service Account using workload-identity.

My problem is that both deployments don't succeed at the same time. The one which runs first succeeds and other fails with following error. Something to do with the gke/GCP credentials.

I get following error

    gcloud pubsub subscriptions describe local-test-v1 --quiet
ERROR: (gcloud.pubsub.subscriptions.describe) You do not currently have an active account selected.
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account:

    $ gcloud config set account ACCOUNT

to select an already authenticated account to use.

Even if I make following changes I don't get it through

gcloud config set account [email protected]

gcloud pubsub subscriptions describe $GCP_SUB_NAME --quiet
    result="$?"
    
    if [ "$result" -ne 0 ]; then
        echo "Subscription not found, exited with non-zero status $result"
        exit $result
    fi

Error I get now

gcloud config set account [email protected]
Updated property [core/account].
+ gcloud pubsub subscriptions describe local-test-v1 --quiet
ERROR: (gcloud.pubsub.subscriptions.describe) Your current active account [[email protected]] does not have any valid credentials
Please run:

  $ gcloud auth login

to obtain new credentials.

For service account, please activate it first:

  $ gcloud auth activate-service-account ACCOUNT

I don't wanna use the GCP client libraries as I want to keep it light weight so either gcloud r curl r the best option.

Can I use gcloud in GKE without the key file?

Can I call googleapis via curl without passing bearer token or how shall I get that in the docker container?

Any ideas... Thanks...

Note#1: workload identity

resource "google_service_account_iam_member" "workload_identity_iam" {
  
member = "serviceAccount:${var.gcp_project}.svc.id.goog[${var.kubernetes_namespace}/${var.kubernetes_service_account_name}]"
   
role   = "roles/iam.workloadIdentityUser"    

service_account_id = google_service_account.sa.name    

depends_on = [google_project_iam_member.pubsub_subscriber_iam, google_project_iam_member.bucket_object_admin_iam]  }

Note#2: GKE SAs

Name:                sa1
Namespace:           some-namespace
Labels:              <none>
Annotations:         iam.gke.io/gcp-service-account: [email protected]
Image pull secrets:  <none>
Mountable secrets:   sa1-token-shj9w
Tokens:              sa1-token-shj9w
Events:              <none>

Name:                sa2  
Namespace:           some-namespace 
Labels:              <none> 
Annotations:         iam.gke.io/gcp-service-account: [email protected] 
Image pull secrets:            <none>
Mountable secrets:   sa2-token-dkhdl 
Tokens:              sa2-token-dkhdl 
Events:              <none>

Note#3: job template for container

apiVersion: batch/v1
kind: Job
metadata:
  namespace: some-namespace
  name: check
  labels:
    helm.sh/chart: check-0.1.0
    app.kubernetes.io/name: check
    app.kubernetes.io/instance: check
    app: check
    app.kubernetes.io/version: "0.1.0"
    app.kubernetes.io/managed-by: Helm
  annotations:
        helm.sh/hook: pre-install,pre-upgrade
        helm.sh/hook-weight: "-4"
spec:
  backoffLimit: 1
  completions: 1
  parallelism: 1
  template:
    metadata:
      name: check
      labels:
        app.kubernetes.io/name: check
        app.kubernetes.io/instance: check
        app: check
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 0
      serviceAccountName: sa1
      securityContext:
        {}
      containers:
        - name: check
          securityContext:
            {}
          image: "eu.gcr.io/some-project/check:500c4166"
          imagePullPolicy: Always
          env:
            # Define the environment variable
            - name: GCP_PROJECT_ID
              valueFrom:
                configMapKeyRef:
                  name: check
                  key: gcpProjectID
            - name: GCP_SUB
              valueFrom:
                configMapKeyRef:
                  name: check
                  key: gcpSubscriptionName
            - name: GCP_BUCKET
              valueFrom:
                configMapKeyRef:         
                  name: check
                  key: gcpBucket
          resources:
            limits:
              cpu: 1000m
              memory: 128Mi
            requests:
              cpu: 100m
              memory: 128Mi

Docker image:

FROM ubuntu:18.04

COPY /checks/pre/ /checks/pre/
ENV HOME /checks/pre/
# Install needed packages
RUN apt-get update && \
    apt-get -y install --no-install-recommends curl \
    iputils-ping \
    tar \
    jq \
    python \
    ca-certificates \
    && mkdir -p /usr/local/gcloud && cd /usr/local/gcloud \
    && curl -o google-cloud-sdk.tar.gz -L -O https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz \
    && tar -xzf google-cloud-sdk.tar.gz \
    && rm -f google-cloud-sdk.tar.gz \
    && ./google-cloud-sdk/install.sh --quiet \
    && mkdir -p /.config/gcloud && chmod 775 -R /checks/pre /.config/gcloud \
    && apt-get autoclean \
    && apt-get autoremove \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin

WORKDIR /checks/pre

USER 1001

ENTRYPOINT [ "/checks/pre/entrypoint.sh" ]