vpc peering not able to communicate between gke and bastion host

Question

I have 2 vpc networks, one consists of a gke cluster (private cluster with private access on subnet) and another vpc with a virtual machine to act as a bastion host for connectivity to the gke cluster.

• I have setup vpc peering between the 2 vpcs and it shows as active

• I have tried creating firewall rules to allow everything on both vpcs, literally allowing all traffic ingress and egress

• I am using IAP tunnel for the bastion host thus have the required firewall set up here for 35.235.240.0/20 and tcp 22

• have tried adding and removing the authorised networks (i.e. the ip of the bastion host to the authorised list) on the cluster

No issues connection to the VM via ssh. It's just when I try to ping the external endpoint of the cluster or kubectl to get nodes from the bastion host that I get error timeout with port 443:

kubectl get nodes -v=10

I1201 09:42:56.605425   68742 loader.go:395] Config loaded from file:  /home/name/.kube/config
I1201 09:42:56.606466   68742 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.28.2 (linux/amd64) kubernetes/89a4ea3" 'https://ip/api?timeout=32s'
I1201 09:43:26.614171   68742 round_trippers.go:508] HTTP Trace: Dial to tcp:ip:443 failed: dial tcp ip:443: i/o timeout
I1201 09:43:26.614238   68742 round_trippers.go:553] GET https://ip/api?timeout=32s  in 30007 milliseconds

Ping also is not successful.

Seems like there is a port block somewhere but I'm not sure where or how given I have allowed everything on both networks (and with the highest priority)

Answer 1

It's normal. Network peering are not transitive. But you have a missing piece in that explanation.

• You have your VPC1 with the bastion
• You have your VPC2 with your GKE control plane
  • That's wrong!! You have a VPCG, a VPC of Google Cloud world which host your GKE control plane, managed by Google.
  • You have a peering between the VPCG and the VPC2.

Therefore you have this network flow:

bastion -> VPC1 -> Peering -> VPC2 -> Peering -> VPCG -> GKE control plane

Because peering is not transitive it fails.

---

Now, what are you options?

• Create a bastion in the VPC2
• Old fashion: create a VPN between VPC1 and VPC2 to solve the peering transitivity issue
• New way: use PSC (private service connect) to abstract the network connectivity and let Google manage the stuff for you. I never tested this newer option.