Unable to make aws sts get-caller-identity from private subnet

Question

I have followed this blog post link https://repost.aws/knowledge-center/s3-instance-access-bucket carefully and it works partially which means if I call from EC2 instance running behind a private subnet (with no Internet Gateway, No route to NAT Gateway, etc) then it does not work at all and I get timed out (Connect timeout on endpoint URL: "https://sts.us-east-1.amazonaws.com/"):-

```
aws sts get-caller-identity --profile Allow_Instance_Profile_EC2_to_Access_S3
```

However, if I run the same aws sts command on another EC2 instance running behind a public subnet then it works like a charm. (Please note that I have attached the same instance profile role to both EC2 instances).

After researching a lot, I found that when we do `aws sts `then it takes a public route to the STS Token service hence it requires a public IP. To overcome this limitation, I then created a VPC Interface Endpoint for STS service as sts.us-east-1.amazonaws.com and attached it to my custom VPC that has only the private subnets thinking that it would resolve the issue if I ran the same command again from the private EC2 instance but again it timed out.

I'm very close to solving this issue and I have done everything from head to toe and really getting frustrated.

Kindly suggest how to solve this issue.

**[Additional details]**
VPC flow logs:-

```
2023-09-07T18:47:44.000+05:30	2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694092664 1694092695 - NODATA

2023-09-07T18:47:45.000+05:30	2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694092665 1694092696 - NODATA

2023-09-07T18:48:05.000+05:30	2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694092685 1694092716 - NODATA

2023-09-07T18:48:14.000+05:30	2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694092694 1694092725 - NODATA

2023-09-07T18:48:27.000+05:30	2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694092707 1694092739 - NODATA

2023-09-07T18:48:40.000+05:30	2 458419607076 eni-0b65a12cfc0782b05 10.0.149.231 10.0.153.8 64356 22 6 56 4896 1694092720 1694092721 ACCEPT OK

2023-09-07T18:48:40.000+05:30	2 458419607076 eni-0b65a12cfc0782b05 10.0.153.8 10.0.149.231 22 64356 6 49 6489 1694092720 1694092721 ACCEPT OK

2023-09-07T18:48:45.000+05:30	2 458419607076 eni-0b65a12cfc0782b05 - - - - - - - 1694092725 1694092756 - NODATA
```

Answer

- First, you need to make sure DNS name is enabled when creating VPC Endpoint
  ![Enter image description here](/media/postImages/original/IMM_pzVjCgQbycXgzcxuSXTg)
- Check your EC2 in private subnet can dns lookup for sts.us-east-1.amazonaws.com, and make sure the IP address from the result is the same in your VPC endpoint
  ![Enter image description here](/media/postImages/original/IMrrEtIs2WQIqAI-IHWLC-jA)
  ![Enter image description here](/media/postImages/original/IMW0qL5rLASICl1GaN6R6Ikg)
- VPC endpoint is just an interface in your VPC, so check your routing, security group to make sure your EC2 can connect to that IP address. You can test by using telnet to VPC endpoint port 443
  ![Enter image description here](/media/postImages/original/IMguotw2SyR7yhnxz01aGqOQ)

Answer

I was able to replicate this environment and calls to STS all worked as expected. Here is what I created:
* VPC, DNS options turned on
* 2 private subnets using default route table and default NACL
* 2 security groups, both wide open (in & out), one for ec2 instance, one for VPC interface endpoints
* One VPC endpoint for STS, both private subnets, using wide open SG
* VPC endpoints for Sessions Manager 
* Deployed an instance into private subnet

This returns two local IP address of the STS VPC endpoints.
```
dig sts.us-east-2.amazonaws.com
```

This works
```
aws sts get-caller-identity
```
Ping does not work. I never expect ping to work on anything.

Can you run a [VPC Reachability Analyzer](https://docs.aws.amazon.com/vpc/latest/reachability/what-is-reachability-analyzer.html) from the instance to the VPC endpoint?

Answer

I have tried this but still timed out:-
`aws sts get-caller-identity --endpoint-url https://sts.us-east-1.amazonaws.com`
I also went through the DNS link https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-support that you provided. I wanted to check why that needs to be enabled in my case because I'm using a custom VPC with only private subnets to run my application. I tried running an EC2 in the public subnet of the default VPC (just to see if my instance profile role is fine or not). I wanted to establish this whole connectivity on my instance running behind a private subnet (of custom VPC).
I would really appreciate it if you could advise on its root cause.

Answer

Okay, few troubleshooting I did based on above suggestions but seems like some firewall is blocking the traffic. Reason being if I try the below command of pingging the sts using `ping sts.us-east-1.amazonaws.com` and behind the scene when I when to VPC and deleted the Interface Endpoint (for STS) then all of a sudden I started getting this desitnation unreachabled ICMP messages:-

```
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1507 Destination Host Unreachable
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1508 Destination Host Unreachable
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1509 Destination Host Unreachable
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1510 Destination Host Unreachable
From ip-10-0-153-8.ec2.internal (10.0.153.8) icmp_seq=1511 Destination Host Unreachable
^C
--- sts.us-east-1.amazonaws.com ping statistics ---
1512 packets transmitted, 0 received, +1311 errors, 100% packet loss, time 1571423ms
pipe 4
```

I then created a new Interface Endpoint for STS and tried again to ping it but still unreachable. I can definitely confirm that DNS Hostnames and DNS resolutions are enabled on this custom VPC (that contains only 2 private subnets).
![Enter image description here](/media/postImages/original/IM0oJLdB_mSiqO7jx_R3PZiA)
![Enter image description here](/media/postImages/original/IMqBW-LD7_S22OcvLO11sJNw)

Also, in the security group of Interface Endpoint of STS, I have allowed all incoming traffic (for time being), refer their screenshots. So it is all whitelisted from Security Group as well. May I know what next I should debug. I feel like I'm close to solving it, tired but do not want to give up on this issue.

![Enter image description here](/media/postImages/original/IMOV3CPAggRJST7CtugvmDUw)
![Enter image description here](/media/postImages/original/IMv4JDzWgkRqy3cQX4JmGzXQ)

Any help would really be appreciated.

Just to recap, I have a custom VPC with 2 Private subnets only (no Internet gateway, no public subnet or no NAT Gateway attached in this VPC). I have attached VPC Gateway (for S3) and Interface Gateway (for STS) in this custom VPC. I'm unable to make call to STS using private EC2 instance excepting the call to traverse via Interface Endpoint(for STS) to STS to get the token. I'm trying to achieve cross-account S3 calls (S3 is located in someother account)  and followed this blog post link https://repost.aws/knowledge-center/s3-instance-access-bucket. The solution is working partially as I can make call to S3 after calling sts from a public EC2 instance (I just provisioned this public instance on a default VPC to test if my instance profile role is working or not) but not from this private EC2 instance (with same instance profile role).
My belief is if I can make call to sts from private EC2 instance then remaining things should work fine as I have already tested my instance profile role from a public instance.

VPC Flow logs:-