Error 401 with cloud scheduler while passing Dataflow template as URL via POST request

Question

I have created a custom template for Dataflow Batch Jobs. Now I need to run every 5 minutes using cloud scheduler. The template is stored in cloud storage. But I'm getting 401 error, whenever I pass the URI of template in my POST request from scheduler

        URI - "https://dataflow.googleapis.com/v1b3/projects/project-id/locations/region/templates:launch?gcsPath=gs://my_bucket/templates/mytemplate 

I'm not sure what the body should contain

Error:

     jsonPayload: {
     targetType: "HTTP"
     @type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
     status: "UNAUTHENTICATED"
     url: "https://storage.cloud.google.com/my_bucket/templates/mydemotemp"
     jobName: "projects/project-id/locations/us-central1/jobs/finaljob"
     }
     httpRequest: {
     status: 401

Can anyone please help as how to go about it. Since I'm new to GCP

Answer 1

It could happen when the service account that you used for Cloud scheduler don't have permission to access to bucket.

You can use your service account for cloud scheduler authentication.

Set service account like below, Or refer here to get detailed guide.

Before setting your cloud scheduler, grant your service account a role like "Storage Object view" or higher level.

Refer here to check GCS roles that required for bucket access.