Why isn't Systems Manager showing my Amazon EC2 instance as a managed instance?

11 minute read

I want to know why I receive an “SSM Agent is not online” or “Connection Lost” error, or why my Amazon Elastic Compute Cloud (Amazon EC2) instance isn’t visible in Fleet Manager.

Short description

You receive the following notifications if your AWS Systems Manager Agent (SSM Agent) is offline on your Amazon Elastic Compute Cloud (Amazon EC2) instance:

Session Manager, a capability of AWS Systems Manager, displays an "SSM Agent is not online. The SSM Agent was unable to connect to a Systems Manager endpoint to register itself with the service" error.
Fleet Manager, a capability of AWS Systems Manager, reports your instance's ping status as a "Connection Lost" error.
Fleet Manager doesn't list your instance.

Note: If your SSM Agent is offline, then your instance doesn't appear as managed in Systems Manager.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Complete the following prerequisites, and then use the AWSSupport-TroubleshootManagedInstance runbook to determine why SSM Agent is offline or can't connect with Systems Manager. For more information, see Run an automation operation powered by Systems Manager Automation and Setting up Automation.

You can also use AWS CLI to troubleshoot your instance, or manually troubleshoot your instance.

Verify that SSM Agent is installed and running

Important: Throughout the troubleshooting steps, select the AWS Region where your instance is located.

Confirm that your Systems Manager supports your operating system (OS). Then, run the command that's specific to your OS to verify that you installed SSM Agent.

Note: The SSM Agent is preinstalled on most Amazon Machine Images (AMI). For more information, see Find AMIs with the SSM Agent preinstalled. If you didn't install SSM Agent in your instance, then manually install it on Linux, Windows, or macOS.

To verify that SSM Agent's status is running, and that SSM Agent isn't in hibernation, complete the following steps:

Open the Amazon EC2 console.
In the left navigation pane, choose Instances.
Select the instance.
Choose Actions, Monitor and troubleshoot, Get system log.

If your SSM Agent is running, then you receive a "Amazon SSM Agent vx.x.x.x is running" message.

If your SSM Agent is in hibernation, then you receive one of the following messages, followed by the reason that your SSM Agent is in hibernation:

"SSM Agent entering hibernation due to error:"
"SSM Agent unable to acquire credentials:"

Note: If you use SSM Agent version 3.3.2471.0 or newer, then review your instance's system log to find the reason that your SSM Agent is in hibernation.

Run the Systems Manager automation runbook

Prerequisite: Before you run the automation, make sure that your AWS Identity and Access Management (IAM) user or role has the necessary permissions. For more information, see the Required IAM permissions section on AWSSupport-TroubleshootManagedInstance.

To run the runbook, complete the following steps:

Open the AWSSupport-TroubleshootManagedInstance runbook.
For the input parameters, enter the following information:
For InstanceId, enter the ID of the affected instance. You can manually enter the instance ID, or you can use the interactive instance picker. If you use the instance picker, then change the filter from Show managed instance only to Show all instances.
(Optional) For AutomationAssumeRole, enter the Amazon Resource Name (ARN) of the IAM role that allows Systems Manager Automation to perform actions on your behalf. If you don't specify a role, then Systems Manager Automation uses the permissions of the user that runs the document.
Choose Execute.

After the automation completes, review the Outputs section. The FinalOutput.Message variable shows whether Systems Manager manages the instance, shows whether a check passed or failed, and provides information about how to troubleshoot a failure.

Use AWS CLI to troubleshoot your offline SSM Agent

Complete the following steps:

Verify that you can access to your instance through SSH or Remote Desktop Protocol.

Run the ssm-cli AWS CLI command to troubleshoot managed instance availability:
Linux & macOS

ssm-cli get-diagnostics --output table

Windows

ssm-cli.exe get-diagnostics --output table

PowerShell

.\ssm-cli.exe get-diagnostics --output table

Note: On Windows Server machines, you must navigate to the C:\Program Files\Amazon\SSM directory before you run the ssm-cli command.

If the SSM Agent isn't functioning correctly, then the ssm-cli command might not be available. For more information, see Troubleshooting SSM Agent.

Manually troubleshoot your Amazon EC2 instance

Verify connectivity to Systems Manager endpoints on port 443

Connectivity verification to Systems Manager endpoints on port 443 is specific to your OS and subnet settings. For a list of Systems Manager endpoints by Region, see Service endpoints.

Note: In the following examples, the ssmmessages endpoint is required for Session Manager.

EC2 Linux instances

Use either Telnet or Netcat commands to verify connectivity to endpoints on port 443 for EC2 Linux instances. Netcat isn't preinstalled on EC2 instances. To manually install Netcat, see Ncat on the Nmap website.

Note: In the following commands, replace RegionID with your instance's Region ID.

Telnet commands:

telnet ssm.RegionID.amazonaws.com 443
telnet ec2messages.RegionID.amazonaws.com 443
telnet ssmmessages.RegionID.amazonaws.com 443

Example Telnet connection:

root@111800186:~# telnet ssm.us-east-1.amazonaws.com 443
Trying 52.46.141.158...
Connected to ssm.us-east-1.amazonaws.com.
Escape character is '^]'.

To exit from telnet, press the Ctrl and ] keys. Enter quit, and then press Enter.

Netcat commands:

nc -vz ssm.RegionID.amazonaws.com 443
nc -vz ec2messages.RegionID.amazonaws.com 443
nc -vz ssmmessages.RegionID.amazonaws.com 443

EC2 Windows instances

To verify connectivity to endpoints on port 443 for EC2 Windows instances, run the following Windows PowerShell commands:

Test-NetConnection ssm.RegionID.amazonaws.com -port 443
Test-NtConnection ec2messages.RegionID.amazonaws.com -port 443
Test-NetConnection ssmmessages.RegionID.amazonaws.com -port 443

Public subnets

Systems Manager endpoints are public. To resolve connectivity issues with instances in a public subnet, your instance's route table must route internet traffic to an internet gateway. Also, your Amazon Virtual Private Cloud (Amazon VPC) security groups and network access control lists (network ACLs) must allow outbound connections on port 443.

Private subnets

Use private IP addresses to privately access Amazon EC2 and Systems Manager APIs. To resolve connectivity issues with an instance in a private subnet, your instance's route table must route internet traffic to a NAT gateway. Or, you must configure your VPC endpoint to reach Systems Manager endpoints.

For more information, see How do I create VPC endpoints so that I can use Systems Manager to manage private EC2 instances without internet access?

Note: Each interface endpoint creates an elastic network interface in the provided subnet.

As a security best practice for private subnets, verify the following settings:

The security group that's attached to your VPC endpoint's network interface allows TCP port 443 inbound traffic from the security group that's attached to your instance.
The security group that's attached to your instance allows TCP port 443 outbound traffic to the private IP address for your VPC endpoint's network interface.

Verify the setup for Default Host Management Configuration

Note: If you didn't activate Default Host Management Configuration, then proceed to the Verify that the correct IAM role is attached to the instance section.

If the IAM role that Default Host Management Configuration created doesn't have sufficient permissions for your use case, then you can add policies.

All the associated instances must use Instance Metadata Service Version 2 (IMDSv2). To check your IMDSv2 configuration, use the MetadataNoToken Amazon CloudWatch metric to determine when there's zero IMDSv1 usage. Then, check if your instances are transitioned to IMDSv2.

Default Host Management Configuration is available in SSM Agent version 3.2.582.0 or later. To verify your SSM Agent version, see Checking the SSM Agent version number.

To verify the setup for Default Host Management Configuration, use either the Systems Manager console or the AWS CLI.

Systems Manager console

Complete the following steps:

Open the Systems Manager console.
In the navigation pane, choose Fleet Manager.
On the Account management dropdown list, choose Default Host Management Configuration.
Verify that the Enable Default Host Management Configuration setting is turned on.

AWS CLI

Run the get-service-setting AWS CLI command to verify the setup for Default Host Management Configuration:

aws ssm get-service-setting \
--setting-id arn:aws:ssm:RegionID:AccountID:servicesetting/ssm/managed-instance/default-ec2-instance-management-role

Note: Replace AccountID with your AWS account ID.

After you activate Default Host Management Configuration, you receive an output that's similar to the following:

{
  "ServiceSetting": {
    "SettingId": "/ssm/managed-instance/default-ec2-instance-management-role",
    "SettingValue": "service-role/AWSSystemsManagerDefaultEC2InstanceManagementRole",
     "LastModifiedDate": 1679492424.738,
    "LastModifiedUser": "arn:aws:sts::012345678910:assumed-role/role/role-name",
    "ARN": "arn:aws:ssm:ap-southeast-1:012345678910:servicesetting/ssm/managed-instance/default-ec2-instance-management-role",
    "Status": "Customized"
  }
}

Note: If the value for SettingValue is $None, then Default Host Management Configuration isn't configured.

Verify that Default Host Management Configuration is using an appropriate IAM role

It's a best practice to use AWSSystemsManagerDefaultEC2InstanceManagementRole IAM when you set up Default Host Management Configuration. To use a different role, make sure to attach the AmazonSSMManagedEC2InstanceDefaultPolicy IAM policy to the role.

If you attached instance profiles to your instances, then remove any permissions that allow the ssm:UpdateInstanceInformation operation. SSM Agent tries to use instance profile permissions before it uses the Default Host Management Configuration permissions. When you allow the ssm:UpdateInstanceInformation operation in your instance profiles, your instance doesn't use the Default Host Management Configuration permissions.

Verify that the correct IAM role is attached to the instance

Note: If you activated Default Host Management Configuration, then proceed to the Verify connectivity to IMDS section.

To make API calls to a Systems Manager endpoint, you must attach the AmazonSSMManagedInstanceCore policy to the IAM role that's attached to your instance. If you're using a custom IAM policy, then confirm that your custom policy uses the permissions in AmazonSSMManagedInstanceCore. Also, make sure that the trust policy for your IAM role allows ec2.amazonaws.com to assume the role. For more information, see Alternative configuration for EC2 instance permissions.

Verify connectivity to IMDS

SSM Agent must communicate with IMDS to get information about your instance. To test the connection, run the following command based on your instance's OS:

Linux

nc -vz 169.254.169.254 80

Windows

Test-NetConnection 169.254.169.254 -port 80

To verify that IMDS is set up for your existing instance, use either the Amazon EC2 console or the AWS CLI.

Amazon EC2 console

Complete the following steps:

Open the Amazon EC2 console.
In the navigation pane, choose Instances, and then select your instance.
Choose Actions, and then choose Instance settings.
Choose Modify instance metadata options.
In the dialog box, make sure that Instance metadata service is Enabled.

AWS CLI

Run the describe-instances AWS CLI command to verify that IMDS is set up for your existing instance:

aws ec2 describe-instances --query "Reservations[*].Instances[*].MetadataOptions" --instance-ids i-012345678910

Example output:

[
  [
    {
      "State": "applied",
      "HttpTokens": "optional",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled",
      "HttpProtocolIpv6": "disabled",
      "InstanceMetadataTags": "disabled"
    }
  ]
]

Note: If the output shows "HttpTokens": "optional", then both IMDSv1 and IMDSv2 are supported. If the output shows "HttpTokens": "required", then only IMDSv2 is supported. If the output shows "HttpEndpoint": "enabled", then IMDS is turned on.

If you're using a proxy on your instance, then the proxy might block connectivity to the metadata URL. To prevent the block, configure your SSM Agent to work with a proxy and set no_proxy for the metadata URL.

To configure SSM Agent to use a proxy, see the following AWS documentation:

Additional troubleshooting

If your instance still doesn't appear as a managed node or shows a lost connection in Systems Manager, then review the SSM Agent logs to continue to troubleshoot. For Linux and macOS, the logs are in /var/log/amazon/ssm. For Windows, the logs are in %PROGRAMDATA%\Amazon\SSM\Logs.

If your instance isn't reporting to SSM Agent, then use Remote Desktop Protocol (RDP) for Windows or SSH for Linux to collect the logs. If you can't collect the logs, then stop your instance and detach the root volume. Then, attach the root volume to another instance in the same Availability Zone as a secondary volume to get the logs.