We are trying to use the Colab Enterprise offering (in Vertex AI) using a shared VPC (hosted in a different project). There is an organizational policy to block external IPs. I have added the Compute Network user permission to the service agent in the Shared VPC Host project, and the runtime template and the runtime are created successfully. But when I try to connect a notebook to the runtime, it tries connecting until a timeout, after which it fails. I checked the runtime logs, this is what I see:
cos.googleapis.com/container_name: "proxy-agent"
message: failed to list pending requests: 401
Your client does not have permission to the requested URL /tun/m/4592f09221234568f8016274df1b36a14/agent/pending
What can be the issue? I guess something networking or IAM related. If I create a runtime in a normal VPC (inside the same project), then the notebook can connect and it's working fine.
Thanks for the report, 401 Error means that the underlying Service Account attached to the Runtime (VM) can't authenticate with the Proxy. But, based on your test, seems to be that the only difference is the Shared VPC vs Local VPC.
Question:
Is there any special DNS configuration for the Shared VPC (Example: DNS for Private Google Access private.googleapis.com/restricted.googleapis.com)
Is this happening to all new Runtimes?
(We made some recent changes to Runtime SA logic in the past 2 weeks)
Is it possible to open a support ticket so we can take a look in detail?