the issue below is an issue i resolved after couple days of research. I expose it so i can have you point of view of how i could have spotted it faster looking at logs etc.
We have an hybrid AWS architecture. And my goal was to connect to redshift from on-premise. The redshift serverless is
- on a VPC lets name it cloud with 3 AZones in 3 subnets : rds1/rds2/rds3.
- has a vpce-endpoint on the subnet rds3 ( why this one ? i don't exactly know )
- attached to a security group where port 5439 is open for corresponding sources.
The VPC cloud :
- has a TransitGateway attachement to 2 subnets tgw1/tgw2 .
Route tables are correctly set.
When i try and connect to redshift; it times out. But when i check for VPC FLOW LOGS
fields @timestamp,@message
| filter dstPort = 5439
| sort @timestamp desc
| limit 20
Here is what i find :
2023-12-18T15:53:12.000+01:00 2 834242179416 eni-xxxxxxxxxxxx xx.xx.xx.xx xx.xx.xx.xx 49801 5439 6 1 52 1702911192 1702911193 ACCEPT OK
so the call to redshift is well routed and accepted but no response never returns.
it appears that according to AWS DOC :
When you attach a VPC to a transit gateway, any resources in Availability Zones where there is no transit gateway attachment cannot reach the transit gateway. If there is a route to the transit gateway in a subnet route table, traffic is forwarded to the transit gateway only when the transit gateway has an attachment in a subnet in the same Availability Zone.
So i added rds3 to the attachment transit gateway and now it works.
but i wonder how i could've seen that directly from logs to identify the problem directly instead of reviewing all security gorups/ NACLS / route tables/ endpoints etc ...
Thank you for your help.
I find that the Reachability Anayzer in the AWS Network Manager is very helpful in finding the source of these kind of issues. You can trace from VPN Gateways all the way to specific IP addresses.
https://docs.aws.amazon.com/vpc/latest/reachability/what-is-reachability-analyzer.html