How do I use SSM Agent logs to troubleshoot issues with SSM Agent in my managed instance?

8 minute read

I want to use my AWS Systems Manager Agent (SSM Agent) logs to troubleshoot issues with SSM Agent.

Short description

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

SSM Agent runs on your managed Amazon Elastic Compute Cloud (Amazon EC2) instance and processes requests from the AWS Systems Manager service. You must meet the following conditions to use SSM Agent. If you don’t meet any of these conditions, then SSM Agent fails to run:

SSM Agent must connect to the required service endpoints.
SSM Agent must have AWS Identity and Access Management (IAM) permissions to call Systems Manager API operations.
Amazon EC2 must retrieve valid credentials from the IAM instance profile. Or, if you configured Default Host Management Configuration, then Amazon EC2 must retrieve credentials from the default role that it provides.

To identify the root cause of the SSM Agent failure, review SSM Agent logs at the following locations:

For Linux:
/var/log/amazon/ssm/amazon-ssm-agent.log
/var/log/amazon/ssm/errors.log
For Windows:
%PROGRAMDATA%\Amazon\SSM\Logs\amazon-ssm-agent.log
%PROGRAMDATA%\Amazon\SSM\Logs\errors.log

Resolution

To use SSM Agent logs to troubleshoot issues, run the ssm-cli command that corresponds with your operating system (OS). Then, complete the following the troubleshooting steps based on your output.

SSM Agent can't reach the metadata service

Systems Manager relies on instance metadata to function correctly. Systems Manager can use either version 1 or version 2 of the Instance Metadata Service (IMDSv1 and IMDSv2) to access instance metadata. Your instance must be able to access 169.254.169.254, which is the IPv4 address of the instance metadata service.

When SSM Agent can't reach the metadata service, it also can't retrieve the AWS Region, IAM role, or instance ID. Error messages similar to the following example indicate that SSM Agent can’t reach the metadata service:

"INFO- Failed to fetch instance ID. Data from vault is empty. RequestError: send request failed caused by: Get http://169.254.169.254/latest/meta-data/instance-id"

This error occurs when you use a proxy for outbound internet connections from your instance before you configure SSM Agent to use a proxy. To resolve this issue, configure SSM Agent to use a proxy.

This error also occurs when you use a custom Amazon Machine Image (AMI) to launch Windows instances with incorrect static network routes. Verify that the route for the metadata service IP points to the correct default gateway. For more information, see How do I troubleshoot the "Waiting for the metadata service" error on my Amazon EC2 Windows instance?

To verify whether metadata is activated for your instance, run the following describe-instances AWS CLI command:

aws ec2 describe-instances --instance-ids example-id --query 'Reservations[*].Instances[*].MetadataOptions'

Note: Replace example-id with your instance ID.

In the following example output, "HttpEndpoint": "enabled" indicates that you didn’t activate metadata for your instance:

“[ [{ "State": "applied", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled", "HttpProtocolIpv6": "disabled", "InstanceMetadataTags": "disabled" }] ]”

If you didn’t activate metadata, then modify your instance metadata options to activate it.

SSM Agent can't reach Systems Manager service endpoints

If SSM Agent can't connect with service endpoints, then SSM Agent can’t communicate with Systems Manager. SSM Agent must make outbound connections to the SSM endpoint ssm.region.amazonaws.com on port 443 to perform Systems Manager API operations. If you must use Session Manager or Run Command, features of AWS Systems Manager, then SSM Agent must also make connections to the ssmmessages.region.amazonaws.com endpoint. For more information about Amazon Virtual Private Cloud (Amazon VPC) configuration requirements for SSM Agent, see Improve the security of EC2 instances by using VPC endpoints for Systems Manager.

Note: SSM Agent retrieves your Region from the instance metadata service and uses it to build the endpoint URL.

When SSM Agent can't connect with the Systems Manager endpoints, you see error messages similar to the following in the SSM Agent logs:

"ERROR [HealthCheck] error when calling AWS APIs. error details - RequestError: send request failed caused by: Post https://ssm.ap-southeast-2.amazonaws.com/: dial tcp [IP_ADDRESS]: i/o timeout"

For instructions on how to resolve this error, see How do I resolve the "RequestError: send request failed caused by:" SSM Agent log error?

If the problem persists, then follow the instructions at: Why isn't Systems Manager showing my Amazon EC2 instance as a managed instance?

SSM Agent doesn't have permissions to call the required Systems Manager API calls

Because SSM Agent isn't authorized to make UpdateInstanceInformation API calls to the service, SSM Agent fails to register itself as online on Systems Manager. For more information, see ssm:* namespace instance-related API operations.

The UpdateInstanceInformation API call must maintain a connection with SSM Agent so that the service knows that SSM Agent correctly functions. SSM Agent calls the Systems Manager service in the cloud every five minutes to provide health check information..

If SSM Agent uses the incorrect IAM permissions, then you see an error that resembles the following example message:

"ERROR [instanceID=i-12345] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::123:assumed-role/123 /i-123456 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:ap-southeast-2:1234567:instance/i-123456 status code: 400, request id: 12345678-1234-1234567 INFO [instanceID=i-1234] [HealthCheck] increasing error count by 1"

If SSM Agent doesn't have any IAM permissions, then you see an error that resembles the following example message:

"ERROR [instanceID=i-1234567] [HealthCheck] error when calling AWS APIs. error details - NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors 2018-05-08 10:58:39 INFO [instanceID=i-1234567] [HealthCheck] increasing error count by 1"

Verify that the IAM role that's attached to the instance includes AmazonSSMManagedInstanceCore AWS managed policy permissions. If the field is blank, then attach an instance profile role and include AmazonSSMManagedInstanceCore permissions.

For more information about the required IAM permissions for Systems Manager, see Additional policy considerations for managed instances.

Systems Manager API call throttling

If multiple managed instances call the UpdateInstanceInformation API operation at the same time, then Systems Manager might throttle those calls.

Error messages similar to the following example indicate that Systems Manager throttled UpdateInstanceInformation API operations for your instance:

"INFO [HealthCheck] HealthCheck reporting agent health. ERROR [HealthCheck] error when calling AWS APIs. error details - ThrottlingException: Rate exceeded status code: 400, request id: 12345-12345-1234 INFO [HealthCheck] increasing error count by 1"

Complete the following troubleshooting steps to prevent "ThrottlingException" errors:

Reduce the frequency of API calls.
If you customized the HealthFrequencyMinutes parameter, then revert the parameter to the default five-minute interval.
Stagger the intervals of API calls so that they don't all run at the same time.

If you still receive “ThrottlingException” errors after you take the previous troubleshooting actions, then request an increase for the utilization quota of your managed nodes. For instructions, see AWS service quotas. For more information, see Service quotas for Managed nodes.

Amazon EC2 can't assume valid credentials from the IAM instance profile

If Amazon EC2 can't assume the IAM role, then you see multiple messages similar to the following example in the SSM Agent logs:

"2023-01-25 09:56:19 ERROR [CredentialRefresher] Retrieve credentials produced error: no valid credentials could be retrieved for ec2 identity"

"2023-01-25 09:56:19 INFO [CredentialRefresher] Sleeping for 1s before retrying retrieve credentials"

If you use IMDSv1 to retrieve metadata from the instance, then you see a message that includes the following example error:

"EC2 cannot assume the role example-instance-profile-name. Please see documentation at https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html#troubleshoot_iam-ec2_errors-info-doc."

It's a best practice to use IMDSv2. However, if you use IMDSv2, then the following command doesn't function:

# curl http://169.254.169.254/latest/meta-data/iam/security-credentials/example=instanceprofile-name

Note: In the previous command example-instance-profile-name is the name of the instance profile.

For more information on how to access instance metadata, see Access instance metadata for an EC2 instance.

To troubleshoot these errors, check the trust policy that's attached to your IAM role. In the policy, specify Amazon EC2 as a service that's allowed to assume the IAM role. The updated policy should look similar to the following example:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": ["ec2.amazonaws.com"] }, "Action": ["sts:AssumeRole"] } ] }

After you update the trust policy, wait for the next automatically scheduled credential refresh. To implement the change immediately, disassociate and then reassociate the instance profile or stop and restart the instance.

To update the trust policy programmatically, use the UpdateAssumeRolePolicy API. For instructions, see The iam/security-credentials/[role-name] document indicates "Code":"AssumeRoleUnauthorizedAccess".