How do I resolve Systems Manager connectivity issues with VPC endpoints in private subnets?

Short description

For Systems Manager to function in private subnets, you must configure your VPC endpoints with DNS settings, security groups, and subnet associations.

If you don't correctly configure your VPC endpoints, then the following issues occur:

• Amazon EC2 instances appear as Offline in the Systems Manager console.
• You get "failed to connect" and "Instance is not connected to Session Manager" errors.

Resolution

Complete the following troubleshooting actions, and then verify that your EC2 instance can connect to Systems Manager.

Check for missing required VPC endpoints and instance permissions

Systems Manager requires the following VPC endpoints to access required AWS solutions:

• Use com.amazonaws.example-region.ssm to allow AWS Systems Manager Agent (SSM Agent) to register with Systems Manager.
• Use com.amazonaws.example-region.ssmmessages to allow Session Manager to connect to managed nodes.
• Use com.amazonaws.example-region.ec2messages to allow SSM Agent to communicate with Systems Manager.

If one or more of the preceding VPC endpoints is missing, then create a VPC endpoint for each missing service. For Service Name, search for and select either ssm, ssmmessages, or ec2messages. For VPC, select your VPC. For Subnets, select the subnet where your instance exists. For Security groups, select your security group.

After you create all three endpoints, SSM Agent automatically detects the new endpoints when they become available. To speed up the process, restart SSM Agent or reboot your instance.

Systems Manager requires instance permission to perform actions on your instances. For more information, see Configure instance permissions required for Systems Manager.

Add subnet associations to VPC endpoints

VPC endpoints must have subnet associations in the same subnets as your instances. If your endpoints don't have subnet associations, then the endpoints can't create the necessary elastic network interfaces.

If your VPC endpoints already exist, then add one subnet to each of the three VPC endpoints.

After you add the subnet to each VPC endpoint, you must wait for the system to create the network interfaces. To speed up the process, reboot your instance. Systems Manager automatically establishes connectivity when SSM Agent reconnects.

Turn on a private DNS name for VPC endpoints

When you deactivate a private DNS name on VPC endpoints, SSM Agent connects with public IP addresses instead of private ones. When you use public IP addresses, SSM Agent connectivity fails in private subnets without a NAT gateway internet access.

Before you turn on a private DNS name on your VPC endpoints, verify that you activated DNS hostnames for your VPC.

After you turn on a private DNS names, you must wait for the VPC endpoint's status to become Available.

Then, run the following commands from your instance to verify your DNS resolution.

ssm:

dig +short ssm.example-region.amazonaws.com

ssmmessages:

dig +short ssmmessages.example-region.amazonaws.com

ec2messages:

dig +short ec2messages.example-region.amazonaws.com

Note: In the preceding commands, replace example-region with your AWS Region.

In the output, verify that the service endpoints resolve to private IP addresses, instead of public IP addresses.

Configure security groups for VPC endpoints

Create a security group for your VPC to allow inbound HTTPS traffic from your instances. For Inbound rules, choose Add rule, and then choose HTTPS for Type and enter the VPC CIDR block for Source.

Then, associate the security group to your VPC endpoints. For Endpoints, select one of the required endpoints. Continue to associate the security group for each of the remaining endpoints.

Verify that your instance's security group allows outbound traffic on port 443 to reach the VPC endpoints. The outbound rule can specify either the VPC CIDR block or the security group on the VPC endpoints.

Resolve custom DNS server configuration issues

If you use custom DNS servers in your VPC, then the DNS servers must resolve VPC endpoint addresses to their private IP addresses. If custom DNS servers forward queries to the internet, then SSM Agent receives public IP addresses and connectivity fails.

To identify DNS resolution issues, use SSH, EC2 Instance Connect, or the EC2 serial console to connect to your instance.

Run the following command to check the DNS servers that your instance uses:

cat /etc/resolv.conf

Then, run the following commands to test the DNS resolution for your Systems Manager endpoints.

ssm:

dig @DNS_Server_IP ssm.example-region.amazonaws.com

ssmmessages:

dig @example-DNS-server-IP ssmmessages.example-region.amazonaws.com

ec2messages:

dig @ example-DNS-server-IP ec2messages.example-region.amazonaws.com

Note: In the preceding commands, replace example-DNS-server-IP with your DNS server's IP address and example-region with your Region.

n the output, verify that the IP addresses are private addresses within your VPC CIDR block, not public IP addresses.

Or, you can configure the Dynamic Host Configuration Protocol (DHCP) option sets for your VPC to use the Amazon DNS server. The Amazon DNS server is the second IP address in your VPC range.

Remove duplicate VPC endpoints

Multiple VPC endpoints that use a private DNS for the same service cause DNS resolution conflicts. To resolve this issue, delete duplicate VPC endpoints. For example, search for the ssm endpoint. If two appear in your search, then delete one. Continue to search for and delete duplicates for the ssmmessages and ec2messages endpoints. To confirm that you deleted the duplicates, check the Endpoints section of the Amazon VPC console.

Confirm that your instance is a managed instance

Complete the following steps:

1. Open the Systems Manager console.
2. In the navigation pane, choose Fleet Manager.
3. Check that your instance appears in the list of managed nodes with a Connection status of Online.
4. If your instance doesn't appear on the list, then see Why isn't Systems Manager showing my Amazon EC2 instance as a managed instance?

To verify that you correctly configured Systems Manager, run the following ssm-cli command:

sudo ssm-cli get-diagnostics --output table

Review the output for failed health checks and missing configuration elements.

Run the following commands to check that your instance can reach the VPC endpoints on port 443.

ssm:

telnet ssm.example-region.amazonaws.com 443

ssmmessages:

telnet ssmmessages.example-region.amazonaws.com 443

ec2messages:

telnet ec2messages.example-region.amazonaws.com 443

Note: In the preceding commands, replace example-region with your Region.

If each command returns a successful connection message, then your instance can reach that service endpoint.

Check your SSM Agent logs for error messages. For more information, see How do I use SSM Agent logs to troubleshoot issues with SSM Agent in my managed instance?

Related information

Complete Session Manager prerequisites