Why is my EC2 instance not displaying as a managed node or showing a "Connection lost" status in Systems Manager?

8 minute read

My Amazon Elastic Compute Cloud (Amazon EC2) instance either lost its connection or isn't displaying under Fleet Manager in the AWS Systems Manager console.

Resolution

A managed instance is an EC2 instance that's used with Systems Manager as a managed node.

To confirm that your EC2 instance meets the prerequisites to be a managed instance, run the AWSSupport-TroubleshootManagedInstance Systems Manager Automation document. Then, verify that your EC2 instance meets the following requirements.

Important: Throughout the troubleshooting steps, select the AWS Region that includes your EC2 instance.

Verify that SSM Agent is installed and running on the instance

After you confirm that your operating system supports Systems Manager, verify that AWS Systems Manager Agent (SSM Agent) is installed and running on your instance.

SSM Agent is preinstalled on some Linux, macOS, and Windows Amazon Machine Images (AMIs). To manually install SSM Agent when the agent isn't preinstalled, see the following documentation:

Linux: Manually installing SSM Agent on EC2 instances for Linux
macOS: Manually installing SSM Agent on EC2 instances for macOS
Windows: Manually installing SSM Agent on EC2 instances for Windows Server

To verify that SSM Agent is running, use operating system-specific commands to check the agent status.

After you finish verifying SSM Agent, run ssm-cli to troubleshoot managed instance availability.

Verify connectivity to Systems Manager endpoints on port 443

To verify connectivity to Systems Manager endpoints on port 443, you must consider your operating system and subnet settings. For a list of Systems Manager endpoints by Region, see AWS Systems Manager endpoints and quotas.

Note: In the examples, the ssmmessages endpoint is required for AWS Systems Manager Session Manager.

EC2 Linux instances

Use either Telnet or Netcat commands to verify connectivity to endpoints on port 443 for EC2 Linux instances.

Note: Replace RegionID with your instance's Region when running commands.

Telnet commands:

telnet ssm.RegionID.amazonaws.com 443
telnet ec2messages.RegionID.amazonaws.com 443
telnet ssmmessages.RegionID.amazonaws.com 443

Telnet connection example:

root@111800186:~# telnet ssm.us-east-1.amazonaws.com 443
Trying 52.46.141.158...
Connected to ssm.us-east-1.amazonaws.com.
Escape character is '^]'.

To exit from telnet, hold down the Ctrl key and press the ] key. Enter quit, and then press Enter.

Netcat commands:

nc -vz ssm.RegionID.amazonaws.com 443
nc -vz ec2messages.RegionID.amazonaws.com 443
nc -vz ssmmessages.RegionID.amazonaws.com 443

Netcat isn't preinstalled on EC2 instances. To manually install Netcat, see Ncat on the Nmap website.

EC2 Windows instances

Use the following Windows PowerShell commands to verify connectivity to endpoints on port 443 for EC2 Windows instances.

Test-NetConnection ssm.RegionID.amazonaws.com -port 443
Test-NetConnection ec2messages.RegionID.amazonaws.com -port 443
Test-NetConnection ssmmessages.RegionID.amazonaws.com -port 443

Public subnets

Systems Manager endpoints are public endpoints. To resolve issues when connecting to an endpoint from an instance in a public subnet, confirm the following points:

Your instance's route table routes internet traffic to an internet gateway.
Your Amazon Virtual Private Cloud (Amazon VPC) security groups and network access control lists (network ACLs) allow outbound connections on port 443.

Private subnets

Use private IP addresses to privately access Amazon EC2 and Systems Manager APIs. To resolve issues when connecting to an endpoint from an instance in a private subnet, confirm one of the following points:

Your instance's route table routes internet traffic to a NAT gateway.
Your VPC endpoint is configured to reach Systems Manager endpoints.

For more information, see How do I create VPC endpoints so that I can use Systems Manager to manage private EC2 instances without internet access?

Note: Each interface endpoint creates an elastic network interface in the provided subnet.

As a security best practice for private subnets, verify the following settings:

The security group attached to your VPC endpoint's network interface allows TCP port 443 inbound traffic from the security group that's attached to your instance.
The security group attached to your instance allows TCP port 443 outbound traffic to the private IP address for your VPC endpoint's network interface.

Verify the setup for Default Host Management Configuration

Note: If you aren't using Default Host Management Configuration, then skip to the Verify that the correct IAM role is attached to the instance section.

Systems Manager automatically manages EC2 instances without an AWS Identity and Access Management (IAM) instance profile when you configure Default Host Management Configuration. When you configure this feature, Systems Manager has permissions to manage all instances in your Region and account. If the permissions aren't sufficient for your use case, add policies to the default IAM role created by Default Host Management Configuration.

All the associated instances must use Instance Metadata Service Version 2 (IMDSv2). To check your IMDSv2 configuration, see When there is zero IMDSv1 usage and Check if your instances are transitioned to IMDSv2.

Default Host Management Configuration is available in SSM Agent version 3.2.582.0 or later. To verify your SSM Agent version, see Checking the SSM Agent version number.

To verify the setup for Default Host Management Configuration, complete the following steps:

Open the Systems Manager console.
In the navigation pane, choose Fleet Manager.
From the Account management dropdown list, choose Default Host Management Configuration.
Verify that the Enable Default Host Management Configuration setting is turned on.

You might also use the following AWS Command Line Interface (AWS CLI) command to verify the setup for Default Host Management Configuration:

Note: Replace AccountID with your AWS account ID when running commands.

aws ssm get-service-setting \
--setting-id arn:aws:ssm:RegionID:AccountID:servicesetting/ssm/managed-instance/default-ec2-instance-management-role

When Default Host Management Configuration is set up, you'll receive a response similar to the following:

{
  "ServiceSetting": {
    "SettingId": "/ssm/managed-instance/default-ec2-instance-management-role",
    "SettingValue": "service-role/AWSSystemsManagerDefaultEC2InstanceManagementRole", 
    "LastModifiedDate": 1679492424.738,
    "LastModifiedUser": "arn:aws:sts::012345678910:assumed-role/role/role-name",
    "ARN": "arn:aws:ssm:ap-southeast-1:012345678910:servicesetting/ssm/managed-instance/default-ec2-instance-management-role",
    "Status": "Customized"
  }
}

Note: If the value for SettingValue is $None, then Default Host Management Configuration isn't configured.

Verify that Default Host Management Configuration is using an appropriate IAM role

The AWSSystemsManagerDefaultEC2InstanceManagementRole role is the recommended IAM role when you set up Default Host Management Configuration. To use a different role, make sure that the role has the AmazonSSMManagedEC2InstanceDefaultPolicy IAM policy attached to it.

If you have instance profiles attached to your EC2 instances, then remove any permissions that allow the ssm:UpdateInstanceInformation operation. SSM Agent tries to use instance profile permissions before using the Default Host Management Configuration permissions. When you allow the ssm:UpdateInstanceInformation operation in your instance profiles, your instance doesn't use the Default Host Management Configuration permissions.

Verify that the correct IAM role is attached to the instance

Note: If you're using Default Host Management Configuration, then skip to the Verify connectivity to IMDS section.

To make API calls to a Systems Manager endpoint, you must attach the AmazonSSMManagedInstanceCore policy to the IAM role that's attached to your instance. If you're using a custom IAM policy, then confirm that your custom policy uses the permissions found in AmazonSSMManagedInstanceCore. Also, make sure that the trust policy for your IAM role allows ec2.amazonaws.com to assume this role.

For more information, see Add permissions to a Systems Manager instance profile (console).

Verify connectivity to IMDS

SSM Agent must communicate with Instance Metadata Service (IMDS) to obtain necessary information about your instance. To test the connection, run the following Netcat command:

nc -vz 169.254.169.254 80

To verify that IMDS is set up for your existing instance, do one of the following steps:

Open the Amazon EC2 console. In the navigation pane, choose Instances, select your instance, and then choose Actions, Instance settings, Modify instance metadata options. In the dialog box, Instance metadata service must be Enabled.

In the AWS CLI, run the describe-instances CLI command.

aws ec2 describe-instances --query "Reservations[*].Instances[*].MetadataOptions" --instance-ids i-012345678910

Output example:

[
  [
    {
      "State": "applied",
      "HttpTokens": "optional",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled",
      "HttpProtocolIpv6": "disabled",
      "InstanceMetadataTags": "disabled"
    }
  ]
]

Note: "HttpTokens": "optional" means both IMDSv1 and IMDSv2 are supported. "HttpTokens": "required" means IMDSv2 is supported. "HttpEndpoint": "enabled" means that IMDS is turned on.

If you're using a proxy on your instance, then the proxy might block connectivity to the metadata URL. To avoid this, make sure that you configure your SSM Agent to work with a proxy and set no_proxy for the metadata URL. To configure SSM Agent to use a proxy, see the following documentation:

Linux: Configure SSM Agent to use a proxy (Linux)
macOS: Configure SSM Agent to use a proxy (macOS)
Windows: Configure SSM Agent to use a proxy for Windows Server instances

Additional troubleshooting

If your instance still doesn't appear as a managed node or shows a lost connection in Systems Manager, then continue troubleshooting in the SSM Agent logs:

Linux and macOS: The SSM Agent logs are in /var/log/amazon/ssm.
Windows: The SSM Agent logs are in %PROGRAMDATA%\Amazon\SSM\Logs.

When your instance isn't reporting to SSM Agent, try signing in using RDP (Windows) or SSH (Linux) to collect the logs. If you can't collect the logs, then you must stop your instance and detach the root volume. Then, attach the root volume to another instance in the same Availability Zone as a secondary volume to obtain the logs.