cloud scheduler on cloud run service permission denied

Question

I'm trying to setup a periodical task using the cloud scheduler to call /task endpoint on my cloud run service.

I'm using the same service account for the scheduler job and cloud run service. The service account has been granted the following IAMs.

Cloud Run Admin
Cloud Run Invoker
Cloud Scheduler Job Runner
Cloud SQL Client
Secret Manager Secret Accessor
Service Account User

{
  "insertId": "",
  "jsonPayload": {
    "status": "PERMISSION_DENIED",
    "url": "https://{CLOUD_RUN_URL}-uc.a.run.app/api/v1/tasks/MyTask",
    "targetType": "HTTP",
    "@type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished",
    "jobName": "projects/{PROJECTID}/locations/us-central1/jobs/MyTask"
  },
  "httpRequest": {
    "status": 403
  },
  "resource": {
    "type": "cloud_scheduler_job",
    "labels": {
      "job_id": "MyTask",
      "project_id": "PROJECT",
      "location": "us-central1"
    }
  },
  "timestamp": "2023-10-23T01:15:43.236434245Z",
  "severity": "ERROR",
  "logName": "projects/{PROJECTID}/logs/cloudscheduler.googleapis.com%2Fexecutions",
  "receiveTimestamp": "2023-10-23T01:15:43.236434245Z"
}

I followed auth to configure my scheduler job to use OCID Token with the same service account and set the audience URL to the task URL.

The Cloud Run service is configured to Allow unauthenticated and Ingress from All. In the logs, I'm seeing the request logged with

{'Host': '{CLOUDRUN_URL}-uc.a.run.app', 'Content-Type': 'application/octet-stream', 'X-Cloudscheduler': 'true', 'X-Cloudscheduler-Jobname': 'MyTask', 'Authorization': '*****', 'Content-Length': '2', 'User-Agent': 'Google-Cloud-Scheduler', 'X-Cloud-Trace-Context': '****', 'Traceparent': '***', 'X-Forwarded-For': '***', 'X-Forwarded-Proto': 'https', 'Forwarded': 'for="***";proto=https', 'Accept-Encoding': 'gzip, deflate, br'}

and got 403.

It seems to me the request is triggered by the scheduler but for some reason, the OCID token from the Authorization header is not authenticated. Any suggestions on where else to look to debug this further?

Answer 1

Turned out it was the CRSF error from the service. I had to exempt the task url from CRSF protection and it works now.

Answer 2

It seems like you enabled ingress to All.

As stated in the Documentation,

A 403 status code can be returned when a service has ingress configured to All, but was blocked due to VPC Service Controls. See the next section on 404 errors for more information on troubleshooting VPC Service Controls denials.