Why isn't Systems Manager showing my Amazon EC2 instance as a managed instance?

10 minute read

I want to resolve “SSM Agent is not online”, “Connection Lost”, or "Instance is not connected to Session Manager" errors, or an Amazon Elastic Compute Cloud (Amazon EC2) instance that isn't a managed instance.

Short description

If your AWS Systems Manager Agent (SSM Agent) is offline, then your instance doesn't appear as managed in Systems Manager.

You receive the following notifications if SSM Agent, is offline on your Amazon EC2 instance:

Session Manager, a capability of AWS Systems Manager, displays an "SSM Agent is not online. The SSM Agent was unable to connect to a Systems Manager endpoint to register itself with the service" error.
Fleet Manager, a capability of AWS Systems Manager, reports your EC2 instance's ping status as a "Connection Lost" error.
Fleet Manager doesn't list your instance.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Complete the following prerequisites and then use the AWSSupport-TroubleshootManagedInstance runbook to determine why SSM Agent is offline or can't connect with Systems Manager. For more information, see Run an automation operation powered by Systems Manager Automation and Setting up Automation.

You can also use AWS CLI to troubleshoot your instance, or manually troubleshoot your instance.

Verify that SSM Agent is installed and running

Important: Throughout the troubleshooting steps, select the AWS Region where your instance is located.

Confirm that your Systems Manager supports your operating system (OS). Then, run the command that's specific to your OS to verify that you installed SSM Agent.

Note: The SSM Agent is preinstalled on most Amazon Machine Images (AMI). For more information, see Find AMIs with the SSM Agent preinstalled. If you didn't install SSM Agent in your instance, then manually install it on Linux, Windows, or macOS.

To verify that SSM Agent's status is running, and that SSM Agent isn't in hibernation, complete the following steps:

Open the Amazon EC2 console.
In the left navigation pane, choose Instances.
Select the instance.
Choose Actions, Monitor and troubleshoot, Get system log.

If your SSM Agent is running, then you receive an "Amazon SSM Agent is running" message.

If SSM Agent is in hibernation, then you receive one of the following messages and the reason that SSM Agent is in hibernation:

"SSM Agent entering hibernation due to error:"
"SSM Agent unable to acquire credentials:"

Note: If you use SSM Agent version 3.3.2471.0 or newer, then review your instance's system log for the reason that SSM Agent is in hibernation.

Run the Systems Manager automation runbook

Prerequisite: Before you run the automation, make sure that your AWS Identity and Access Management (IAM) user or role has the necessary permissions. For more information, see the Required IAM permissions section on AWSSupport-TroubleshootManagedInstance.

To run the runbook, complete the following steps:

Open the AWSSupport-TroubleshootManagedInstance runbook.
For the input parameters, enter the following information:
For InstanceId, enter your instance ID. You can manually enter the instance ID, or you can use the interactive instance picker. If you use the instance picker, then change the filter from Show managed instance only to Show all instances.
(Optional) For AutomationAssumeRole, enter the Amazon Resource Name (ARN) of the IAM role that allows Systems Manager Automation to perform actions on your behalf. If you don't specify a role, then Systems Manager Automation uses the permissions of the user that runs the document.
Choose Execute.

After the automation completes, review the Outputs section. The FinalOutput.Message variable shows whether Systems Manager manages the instance, shows whether a check passed or failed, and provides information about how to troubleshoot a failure.

Use AWS CLI to troubleshoot your offline SSM Agent

Complete the following steps:

Verify that you can access to your instance through SSH or Remote Desktop Protocol.

Run the ssm-cli command to troubleshoot managed instance availability:
For Linux & macOS:

ssm-cli get-diagnostics --output table

For Windows:

ssm-cli.exe get-diagnostics --output table

For PowerShell:

.\ssm-cli.exe get-diagnostics --output table

Note: On Windows Server machines, you must navigate to the C:\Program Files\Amazon\SSM directory before you run the ssm-cli command.

If the SSM Agent isn't functioning correctly, then the ssm-cli command might not be available. For more information, see Troubleshooting SSM Agent.

Manually troubleshoot your Amazon EC2 instance

Verify connectivity to Systems Manager endpoints on port 443

Check that your instance can connect to the required Systems Manager endpoints on port 443.

Note: In the following examples, the ssmmessages endpoint is required for Session Manager.

EC2 Linux instances

Use either Telnet or Netcat commands to verify connectivity to endpoints on port 443 for EC2 Linux instances. Netcat isn't preinstalled on EC2 instances. To manually install Netcat, see Ncat on the Nmap website.

Note: In the following commands, replace RegionID with your instance's Region ID.

Telnet commands:

telnet ssm.RegionID.amazonaws.com 443
telnet ec2messages.RegionID.amazonaws.com 443
telnet ssmmessages.RegionID.amazonaws.com 443

Example Telnet output:

root@111800186:~# telnet ssm.us-east-1.amazonaws.com 443
Trying 52.46.141.158...
Connected to ssm.us-east-1.amazonaws.com.
Escape character is '^]'.

To exit from telnet, press the Ctrl and ] keys. Enter quit, and then press Enter.

Netcat commands:

nc -vz ssm.RegionID.amazonaws.com 443
nc -vz ec2messages.RegionID.amazonaws.com 443
nc -vz ssmmessages.RegionID.amazonaws.com 443

EC2 Windows instances

To verify connectivity to endpoints on port 443 for EC2 Windows instances, run the following Windows PowerShell commands:

Test-NetConnection ssm.RegionID.amazonaws.com -port 443
Test-NtConnection ec2messages.RegionID.amazonaws.com -port 443
Test-NetConnection ssmmessages.RegionID.amazonaws.com -port 443

Public subnets

Systems Manager endpoints are public. To resolve connectivity issues with instances in a public subnet, your instance's route table must route internet traffic to an internet gateway. Also, your Amazon Virtual Private Cloud (Amazon VPC) security groups and network access control lists (network ACLs) must allow outbound connections on port 443.

Private subnets

Use private IP addresses to privately access Amazon EC2 and Systems Manager APIs. To resolve connectivity issues with an instance in a private subnet, your instance's route table must route traffic to a NAT gateway. Or you must configure your VPC endpoint to reach Systems Manager endpoints.

For more information, see How do I create VPC endpoints so that I can use Systems Manager to manage private EC2 instances without internet access?

Note: Each interface endpoint creates an elastic network interface in the provided subnet.

As a security best practice for private subnets, verify the following settings:

The security group attached to your VPC endpoint's network interface allows TCP port 443 inbound traffic from the security group that's attached to your instance.
The security group attached to your instance allows TCP port 443 outbound traffic to the private IP address for your VPC endpoint's network interface.

Verify the setup for Default Host Management Configuration

Note: If you didn't activate Default Host Management Configuration, then proceed to the Verify that the correct IAM role is attached to the instance section.

If the IAM role that Default Host Management Configuration created doesn't have sufficient permissions for your use case, then you can add policies.

All the associated instances must use Instance Metadata Service Version 2 (IMDSv2). To check your IMDSv2 configuration, use the MetadataNoToken Amazon CloudWatch metric to determine if any processes use IMDSv1. If any of your processes use IMDSv2, then, transition your instances to IMDSv2.

Default Host Management Configuration is available in only SSM Agent version 3.2.582.0 or later. To verify your SSM Agent version, see Checking the SSM Agent version number.

To verify the setup for Default Host Management Configuration, use either the Systems Manager console or the AWS CLI.

Systems Manager console

Complete the following steps:

Open the Systems Manager console.
In the navigation pane, choose Fleet Manager.
On the Account management dropdown list, choose Default Host Management Configuration.
Verify that the Enable Default Host Management Configuration setting is turned on.

AWS CLI

Run the get-service-setting AWS CLI command to verify the setup for Default Host Management Configuration:

aws ssm get-service-setting \
--setting-id arn:aws:ssm:RegionID:AccountID:servicesetting/ssm/managed-instance/default-ec2-instance-management-role

Note: Replace AccountID with your AWS account ID.

After you activate Default Host Management Configuration, you receive an output that's similar to the following:

{
  "ServiceSetting": {
    "SettingId": "/ssm/managed-instance/default-ec2-instance-management-role",
    "SettingValue": "service-role/AWSSystemsManagerDefaultEC2InstanceManagementRole",
     "LastModifiedDate": 1679492424.738,
    "LastModifiedUser": "arn:aws:sts::012345678910:assumed-role/role/role-name",
    "ARN": "arn:aws:ssm:ap-southeast-1:012345678910:servicesetting/ssm/managed-instance/default-ec2-instance-management-role",
    "Status": "Customized"
  }
}

Note: If the value for SettingValue is $None, then Default Host Management Configuration isn't configured.

Verify that Default Host Management Configuration is using an appropriate IAM role

It's a best practice to use AWSSystemsManagerDefaultEC2InstanceManagementRole IAM when you set up Default Host Management Configuration. To use a different role, make sure to attach the AmazonSSMManagedEC2InstanceDefaultPolicy IAM policy to the role.

If you attached instance profiles to your instances, then remove any permissions that allow the ssm:UpdateInstanceInformation operation. SSM Agent tries to use instance profile permissions before it uses the Default Host Management Configuration permissions. When you allow the ssm:UpdateInstanceInformation operation in your instance profiles, your instance doesn't use the Default Host Management Configuration permissions.

Verify that the correct IAM role is attached to the instance

Note: If you activated Default Host Management Configuration, then proceed to the Verify connectivity to IMDS section.

To make API calls to a Systems Manager endpoint, you must attach the AmazonSSMManagedInstanceCore policy to the IAM role that's attached to your instance. If you're using a custom IAM policy, then confirm that your custom policy uses the permissions in AmazonSSMManagedInstanceCore. Also, make sure that the trust policy for your IAM role allows ec2.amazonaws.com to assume the role. For more information, see Alternative configuration for EC2 instance permissions.

Verify connectivity to IMDS

SSM Agent must communicate with IMDS to get information about your instance. To test the connection, run the following command based on your instance's OS:

For Linux:

nc -vz 169.254.169.254 80

For Windows:

Test-NetConnection 169.254.169.254 -port 80

To verify that IMDS is set up for your existing instance, use either the Amazon EC2 console or the AWS CLI.

Amazon EC2 console

Complete the following steps:

Open the Amazon EC2 console.
In the navigation pane, choose Instances, and then select your instance.
Choose Actions then choose Instance settings.
Choose Modify instance metadata options.
In the dialog box, make sure that Instance metadata service is Enabled.

AWS CLI

Run the describe-instances AWS CLI command to verify that IMDS is set up for your existing instance:

aws ec2 describe-instances --query "Reservations[*].Instances[*].MetadataOptions" --instance-ids i-012345678910

Example output:

[
  [
    {
      "State": "applied",
      "HttpTokens": "optional",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled",
      "HttpProtocolIpv6": "disabled",
      "InstanceMetadataTags": "disabled"
    }
  ]
]

Note: If the output shows "HttpTokens": "optional", then your configuration supports both IMDSv1 and IMDSv2. If the output shows "HttpTokens": "required", then your configuration supports only IMDSv2. If the output shows "HttpEndpoint": "enabled", then IMDS is turned on.

If you use a proxy on your instance, then the proxy blocks connectivity to the metadata URL. To prevent this issue, configure your SSM Agent to work with a proxy and set no_proxy for the metadata URL.

Troubleshoot other issues

Check your SSM Agent logs for error messages. For information about troubleshooting with SSM Agent logs, see How do I use SSM Agent logs to troubleshoot issues with SSM Agent in my managed instance?

If your instance doesn't report to SSM Agent, then use Remote Desktop Protocol (RDP) for Windows or SSH for Linux to collect the logs. If you can't collect the logs, then stop your instance and detach the root volume. Then, attach the root volume to another instance in the same Availability Zone as a secondary volume to get the logs.