I've been trying to move my application from GCP cloud run to GKE. I am using a finetuned language model trained using vertex ai and the workload when deployed on the cluster keeps throwing crashloop error. When I go through the logs, I can see the service account doesn't have permission to access aiplatform.models.get.
I've provided all the required permission to this service account used by the cluster and it still persist. Error:
metadata {
key: "permission"
value: "aiplatform.models.get"
}
]
[2024-02-09 19:06:54 +0000] [7] [INFO] Worker exiting (pid: 7)
I've updated the permission for the service account using UI as well as this command:
gcloud projects add-iam-policy-binding <PROJECT_ID> --member=serviceAccount:<service-account-name>@<PROJECT_ID>.iam.gserviceaccount.com --role=roles/aiplatform.viewer
note: I've tried adding aiplatform.admin and editor role but the issue continue to persist.