Unable to use aws ssm start-session cli commands

Question

I have been using aws ssm start-session cli commands to connect to ec2 instances and portforward to private resources inside my vpc for months now, but suddenly yesterday they stopped working. The start-session command would just hang for 10s of minutes and finally return an error like this

```
Connect timeout on endpoint URL: "https://ec2.us-west-2.amazonaws.com/"
```
As I have been trying to figure out what has changed I feel confident that there is some region issue with the `ssmmessages.us-west-2.amazonaws.com` domain. My coworkers in different states are able to connect like normal without issue but I am never able to.

when I dig that domain I get `52.94.185.41` and when I try to connect to that domain with curl it hangs up and eventually times out at 90% of the time. Every once in a while it will return a 400 response (any response is a good thing in this case) but most often it just hangs. 
```
curl -v  https://ssmmessages.us-west-2.amazonaws.com/
* Host ssmmessages.us-west-2.amazonaws.com:443 was resolved.
* IPv6: (none)
* IPv4: 52.94.185.41
*   Trying 52.94.185.41:443...
* connect to 52.94.185.41 port 443 from 192.168.0.26 port 50068 failed: Connection timed out
* Failed to connect to ssmmessages.us-west-2.amazonaws.com port 443 after 135035 ms: Couldn't connect to server
* Closing connection
curl: (28) Failed to connect to ssmmessages.us-west-2.amazonaws.com port 443 after 135035 ms: Couldn't connect to server
```

I have tried this on 2 different residential networks and even spoken to my ISP about it when i suspected that there could be something going on there. The only thing I've been able to do to workaround this is use a mobile hotspot which allows me to connect to `ssmmessages.us-west-2.amazonaws.com` just fine every time and the start-session command works every time. However, this is not a real workaround because throughput is terrible and I still can't really do my work.

As I've been trying to understand what is going on here I feel like this is some regional IP issue or edge routing that is degraded. I don't have a business level support plan so this is my only place I can reach out reporting the issue afaik.

Any

Gideon · Answer

seems like the fix was waiting 48 hrs. :shrug:

re:Post Agent · Answer

It sounds like you're experiencing connectivity issues with the AWS Systems Manager Session Manager service. Based on your description, the problem appears to be related to network connectivity between your residential networks and the AWS SSM endpoints, specifically `ssmmessages.us-west-2.amazonaws.com`.

Here are some troubleshooting steps you can take:

1. **Check your security group and VPC settings**: Ensure they allow HTTPS (port 443) outbound traffic to the Systems Manager endpoints in your region, including `ssm.us-west-2.amazonaws.com`, `ec2messages.us-west-2.amazonaws.com`, and `ssmmessages.us-west-2.amazonaws.com`.

2. **Investigate network connectivity issues**: Since you've tried multiple residential networks with the same result, but a mobile hotspot works, this suggests there might be routing issues with your ISP or some network configuration blocking access to the AWS endpoints. This could be due to firewall restrictions, routing problems, or other network connectivity issues.

3. **Run diagnostic tools**: If you can connect to your instance through other means (like SSH or EC2 Instance Connect), you can run the SSM CLI diagnostic tool to gather more information:
```
sudo ssm-cli get-diagnostics --output table
```
This will help identify any missing Session Manager prerequisites.

4. **Check for regional IP routing issues**: Since your coworkers in different states can connect normally, this strengthens the theory that there might be a regional routing issue between your location and the AWS endpoints.

5. **Try using a different AWS region**: As a temporary workaround, you could try connecting to instances in a different AWS region if that's an option for your workload.

6. **Contact your ISP again**: With the specific information about the endpoint that's failing to connect, your ISP might be able to investigate further if there are any routing issues to that specific IP range.

Since you've already verified the issue occurs across multiple residential networks but works on a mobile connection, this strongly suggests an ISP or regional network routing issue rather than a problem with your AWS configuration or the Session Manager service itself.  
**Sources**  
[Restore connection to EC2 instance | AWS re:Post](https://repost.aws/knowledge-center/ec2-restore-connection)  
[Ping status: Connection lost | AWS re:Post](https://repost.aws/questions/QUKs5TVcZuRDOIs8B2c9piNA/ping-status-connection-lost)

Unable to use aws ssm start-session cli commands

Add your answer

Relevant content