I have followed this blog post link https://repost.aws/knowledge-center/s3-instance-access-bucket carefully and it works partially which means if I call from EC2 instance running behind a private subnet (with no Internet Gateway, No route to NAT Gateway, etc) then it does not work at all and I get timed out (Connect timeout on endpoint URL: "https://sts.us-east-1.amazonaws.com/"):-
aws sts get-caller-identity --profile Allow_Instance_Profile_EC2_to_Access_S3
However, if I run the same aws sts command on another EC2 instance running behind a public subnet then it works like a charm. (Please note that I have attached the same instance profile role to both EC2 instances).
After researching a lot, I found that when we do aws sts then it takes a public route to the STS Token service hence it requires a public IP. To overcome this limitation, I then created a VPC Interface Endpoint for STS service as sts.us-east-1.amazonaws.com and attached it to my custom VPC that has only the private subnets thinking that it would resolve the issue if I ran the same command again from the private EC2 instance but again it timed out.
I'm using a custom VPC with only private subnets to run my application. I tried running an EC2 in the public subnet of the default VPC (just to see if my instance profile role is fine or not). I wanted to establish this whole connectivity on my instance running behind a private subnet (of custom VPC). I would really appreciate it if you could advise on its root cause.
Following is the configuration of STS Interface Endpoint:-
I'm very close to solving this issue and I have done everything from head to toe and really getting frustrated.
Kindly suggest how to solve this issue.
[Additional Details]
Few troubleshooting I did based on suggestions but seems like some firewall is blocking the traffic. Reason is if I try this command ping sts.us-east-1.amazonaws.com and behind the scene when I go to VPC and deleted the Interface Endpoint (for STS) then all of a sudden I started getting this destination unreachable ICMP messages:-
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1507 Destination Host Unreachable
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1508 Destination Host Unreachable
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1509 Destination Host Unreachable
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1510 Destination Host Unreachable
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1511 Destination Host Unreachable
^C
--- sts.us-east-1.amazonaws.com ping statistics ---
1512 packets transmitted, 0 received, +1311 errors, 100% packet loss, time 1571423ms
pipe 4
I then created a new Interface Endpoint for STS and tried again to ping it but still gets timed out eventually. I can definitely confirm that DNS Hostnames and DNS resolutions are enabled on this custom VPC (that contains only 2 private subnets).
Also, in the security group of Interface Endpoint of STS, I have allowed all incoming traffic (for time being), refer their screenshots. [![enter image description here][4]][4] [![enter image description here][5]][5]
So it is all whitelisted from the Security Group as well. May I know what next I should debug? I feel like I'm close to solving it, tired but do not want to give up on this issue.
Any help would really be appreciated.
[Added Network Reachability Analyzer] -> Reachable as per the screenshots but ping to ping sts.us-east-1.amazonaws.com still timed out if manually done from the EC2 instance (refer to screenshot)
[New VPC Logs while doing the Reachability Analyzer):-
eni-0b65a12cfc0782b05 (of EC2 Instance Connect Endpoint)
2023-09-12T08:42:21.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488341 1694488372 - NODATA
2023-09-12T08:42:27.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488347 1694488378 - NODATA
2023-09-12T08:42:30.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 10.0.149.231 10.0.153.8 39871 22 6 6 448 1694488350 1694488352 ACCEPT OK
2023-09-12T08:42:30.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 10.0.153.8 10.0.149.231 22 39871 6 4 280 1694488350 1694488352 ACCEPT OK
2023-09-12T08:42:55.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488375 1694488406 - NODATA
2023-09-12T08:42:56.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488376 1694488407 - NODATA
2023-09-12T08:42:59.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488379 1694488410 - NODATA
2023-09-12T08:43:27.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488407 1694488439 - NODATA
2023-09-12T08:43:31.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 10.0.149.231 10.0.153.8 39871 22 6 6 448 1694488411 1694488412 ACCEPT OK
2023-09-12T08:43:31.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 10.0.153.8 10.0.149.231 22 39871 6 4 280 1694488411 1694488412 ACCEPT OK
2023-09-12T08:43:55.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488435 1694488466 - NODATA
2023-09-12T08:43:56.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488436 1694488467 - NODATA
2023-09-12T08:43:59.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488439 1694488470 - NODATA
2023-09-12T08:44:20.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488460 1694488492 - NODATA
2023-09-12T08:44:28.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694488468 1694488499 - NODATA
2023-09-12T08:44:30.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 10.0.149.231 10.0.153.8 39871 22 6 6 448 1694488470 1694488472 ACCEPT OK
2023-09-12T08:44:30.000+05:30 2 458419607076 eni-0b65a12cfc0782b05 10.0.153.8 10.0.149.231 22 39871 6 4 280 1694488470 1694488472 ACCEPT OK
eni-07523ca7c37243b2b (of Private EC2 Instance)
2023-09-12T08:40:46.000+05:30 2 458419607076 eni-07523ca7c37243b2b 10.0.149.231 10.0.153.8 39871 22 6 55 4816 1694488246 1694488396 ACCEPT OK
2023-09-12T08:40:46.000+05:30 2 458419607076 eni-07523ca7c37243b2b 10.0.153.8 10.0.149.231 22 39871 6 45 6241 1694488246 1694488396 ACCEPT OK
2023-09-12T08:43:46.000+05:30 2 458419607076 eni-07523ca7c37243b2b 10.0.153.8 10.0.149.231 22 39871 6 8 560 1694488426 1694488516 ACCEPT OK
2023-09-12T08:43:46.000+05:30 2 458419607076 eni-07523ca7c37243b2b 10.0.149.231 10.0.153.8 39871 22 6 12 896 1694488426 1694488516 ACCEPT OK
eni-0e2793b7226a717d6(of STS Interface Endpoint)
2023-09-12T08:40:43.000+05:30 2 458419607076 eni-0e2793b7226a717d6 - - - - - - - 1694488243 1694488315 - NODATA
2023-09-12T08:41:04.000+05:30 2 458419607076 eni-0e2793b7226a717d6 - - - - - - - 1694488264 1694488343 - NODATA
2023-09-12T08:41:17.000+05:30 2 458419607076 eni-0e2793b7226a717d6 - - - - - - - 1694488277 1694488356 - NODATA
eni-0dc4385564f4a3814 (of STS Interface Endpoint)
2023-09-12T08:40:25.000+05:30 2 458419607076 eni-0dc4385564f4a3814 - - - - - - - 1694488225 1694488320 - NODATA
2023-09-12T08:40:27.000+05:30 2 458419607076 eni-0dc4385564f4a3814 - - - - - - - 1694488227 1694488306 - NODATA
2023-09-12T08:41:04.000+05:30 2 458419607076 eni-0dc4385564f4a3814 - - - - - - - 1694488264 1694488350 - NODATA
As per the logs, I can say that the request is not reaching two ENI's of the STS Interface Endpoints from my private EC2 instance but the reachability analyzer is saying as reachable which is a false alarm.
You need to enable Private DNS names in the Interface Endpoint, otherwise you need to specify the custom DNS name created for that endpoint like this
aws sts get-caller-identity --endpoint-url https://vpce-xxxxxxxxx.sts.us-east-1.vpce.amazonaws.com.When Private DNS names is enabled it looks like this: