GCP workload identity federation - Github provider - 'Unable to acquire impersonated credentials'

Question

I've followed these instructions to the letter to allow me to use the short lived token authentication method to access gcloud resources from our github actions workflow.

I've created the service account, workload identity pool and github provider pool using the exact instructions above, but it doesn't appear that the auth step is getting the correct token (or any token at all). The GCP service account has the correct IAM permissions.

On the gcloud compute instances list step, I'm receiving the error:

ERROR: (gcloud.compute.instances.list) There was a problem refreshing your current auth tokens: ('Unable to acquire impersonated credentials: No access token or invalid expiration in response.', '{\n  "error": {\n    "code": 403,\n    "message": "The caller does not have permission",\n    "status": "PERMISSION_DENIED"\n  }\n}\n')
Please run:

  $ gcloud auth login

to obtain new credentials.

My github actions file is as follows:

jobs:
  backup:
    permissions:
      contents: 'read'
      id-token: 'write'
    runs-on: ubuntu-latest
    steps:
      - name: 'Checkout code'
        uses: actions/checkout@v3

      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@v0'
        with:
          workload_identity_provider: 'projects/*******/locations/global/workloadIdentityPools/**REDACTED**/providers/github-provider'
          service_account: '**REDACTED**@**REDACTED**.iam.gserviceaccount.com'

      # Install gcloud, `setup-gcloud` automatically picks up authentication from `auth`.
      - name: 'Set up Cloud SDK'
        uses: 'google-github-actions/setup-gcloud@v0'

      - name: 'Use gcloud CLI'
        run: gcloud compute instances list

I enabled logging for the token exchange and I can see it occurring (with no obvious errors) in GCP logs either. So I'm completely stumped.

Any ideas?

Answer 1

So I later found out what this was. Despite running:

gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_EMAIL \
    --role=roles/iam.workloadIdentityUser \
    --member="MEMBER_EXPRESSION"

As per the docs, it had not granted permission - I went into the console and checked the workload identity pool under "connected service accounts" menu (to the left) and the service account wasn't in there, so I added it manually.

Answer 2

In addition to the OP's own answer of the service account not being connected (bound) at all, this can result from the service account binding being constrained using attribute mappings.

In the default setup for GitHub Actions discussed here on the google blog for WIF, the provider is set up with a set of attribute mappings:

--attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository"

Those attributes can be used to restrict access when you connect (bind) the service account... in the following, the member uses the repository attribute to constrain it so that only actions executing in my-org/my-repo on GitHub will be permitted.

gcloud iam service-accounts add-iam-policy-binding "my-service-account@${PROJECT_ID}.iam.gserviceaccount.com" \
  --project="${PROJECT_ID}" \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/1234567890/locations/global/workloadIdentityPools/my-pool/attribute.repository/my-org/my-repo"

A cross-repository provider

Of course, you want to use strong restrictions here. Restricting to a repository, or even a particular branch (so only main branch actions have privilege to deploy to production, for example). No restrictions allows anything, which is absolutely not what you want!!!

In my case, I set up my WIF provider, then tried to reuse it from another repository, resulting in the error experienced by the OP.

I chose to add the repository_owner attribute mapping from the list of all possible attributes that are in GitHub's OIDC token (the attribute mappings are editable in the google cloud console), then bind my service account to that rather than the repository-specific principal:

--attribute-mapping="google.subject=assertion.sub,attribute.repository_owner=assertion.repository_owner"

and

gcloud iam service-accounts add-iam-policy-binding "my-service-account@${PROJECT_ID}.iam.gserviceaccount.com" \
  --project="${PROJECT_ID}" \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/1234567890/locations/global/workloadIdentityPools/my-pool/attribute.repository_owner/my-org"

Bingo, it works a charm now.

Take care to think about your attack surface though, loosening this constraint too widely creates real vulnerability.

Answer 3

I found all of the answers here a bit cryptic. I don'd use the cli to interact with GCP, I usually use the UI and Terraform to config things. What I realised I was missing here was that I'd created the identity pool

But I'd not mapped that pool to the service account user:

So ensure you've added the service account to the identity pool

This is pretty much the same as this answer but with pictures of what to look for.