How do I troubleshoot permissions errors and integration failures between Session Manager and CloudWatch logs?

Short description

The following issues cause integration failures between Session Manager and CloudWatch Logs:

• You didn't configure your encryption settings.
• AWS Systems Manager Agent (SSM Agent) or your Amazon Elastic Compute Cloud (Amazon EC2) instance profile doesn't have the necessary AWS Identity and Access Management (IAM) permissions.
  Note: If your instance doesn't have the necessary IAM permissions, then Session Manager creates a new log stream for each session.
• Your instances in a private subnet can't access the virtual private cloud (VPC) endpoint.
• Your instance can't connect to CloudWatch Logs.

Resolution

Identify the issue

View your SSM Agent logs

To Identify issues that cause integration failures, use Session Manager to connect to your instance, and then view your SSM Agent logs.

In your logs, look for the following CloudWatch Logs errors:

• "An error occurred (AccessDeniedException) when calling the DescribeLogGroups operation." permissions errors show that your IAM doesn't have the necessary permissions.
• Your instance can't connect to CloudWatch Logs endpoints.

Troubleshoot your issue based on the error that you identify in your SSM Agent logs.

Note: If CloudWatch Logs stops access to your SSM Agent logs, then temporarily deactivate CloudWatch Logs. After you troubleshoot the issue, reactivate CloudWatch Logs.

Activate debug logging

To view more detailed information in your SSM Agent logs, complete the following steps

1. To open your SSM Agent configuration file, run the following command:

   sudo vi /etc/amazon/ssm/amazon-ssm-agent.json

2. In the configuration file, set the LogLevel value to debug:

   "LogLevel": "debug"

3. Save your SSM Agent configuration file.

4. Run one of the following commands to restart SSM Agent.
   For Linux:

   sudo systemctl restart amazon-ssm-agent

   For Windows:

   Restart-Service AmazonSSMAgent

Check that you encrypted your CloudWatch log group

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

If you get the following error message after you tried to start a session, then you didn't encrypt your CloudWatch log group:

"We couldn't start the session because encryption is not set up on the selected CloudWatch log group"

If you activate Allow only encrypted CloudWatch log groups in your Systems Manager preferences, then you must encrypt the log group with an AWS Key Management Service (AWS KMS) key. Or, deactivate the encryption requirement.

To turn off the encryption requirement in Session Manager preferences, see Logging session data using Amazon CloudWatch Logs (console). In the preferences, clear Allow only encrypted CloudWatch log groups, and then save your changes.

Resolve IAM permissions issues

Your instance IAM role must have permissions to interact with CloudWatch Logs. If you receive a "log stream does not exist" error when you start a Session Manager session, then the role doesn't have the necessary permissions.

If you configure Session Manager to send logs to both CloudWatch Logs and Amazon Simple Storage Service (Amazon S3) with encryption activated, then your role requires additional permissions.

If you use a managed node in a hybrid and multicloud environment, then you must use the IAM service role that you used for the hybrid activation.

Resolve missing IAM permissions for the instance profile

To resolve missing IAM permissions for the instance profile, add the following permissions to the policy that's attached to the IAM role:

{  
 "Version": "2012-10-17",  
 "Statement": [  
  {  
   "Effect": "Allow",  
   "Action": [  
    "logs:CreateLogGroup",  
    "logs:CreateLogStream",  
    "logs:PutLogEvents",  
    "logs:DescribeLogStreams",  
    "logs:DescribeLogGroups"  
   ],  
   "Resource": "*"  
  }  
 ]  
}  

Or, attach the CloudWatchAgentServerPolicy AWS managed policy to the instance profile role. The policy includes the necessary CloudWatch Logs permissions. For more information, see the Policy: CloudWatchAgentServerPolicy section of Additional policy considerations for managed instances.

If the IAM role doesn't exist, then create a role with permissions for CloudWatch Logs.

After you add the required permissions, restart SSM Agent.

Resolve Amazon S3 encryption configuration permissions for the instance profile

If you configured the following settings, then the instance profile must have permission to perform GetEncryptionConfiguration on your S3 bucket:

• You activated encryption for Session Manager.
• Session manager sends logs to CloudWatch Logs.
• Session Manager sends logs to an S3 bucket.

Add the following permission to the IAM role's policy:

{  
 "Effect": "Allow",  
 "Action": [  
  "s3:GetEncryptionConfiguration"  
 ],  
 "Resource": "*"  
}

Resolve KMS permissions for encrypted log groups

If you use encrypted CloudWatch log groups or S3 buckets with customer managed KMS keys, then your role must have the required KMS key permissions.

Note: If you use only AWS managed keys, then don't add KMS permissions to the IAM role's policy.

Add the following KMS permissions to the role's policy:

{  
 "Effect": "Allow",  
 "Action": [  
  "kms:Decrypt",  
  "kms:GenerateDataKey",  
  "kms:DescribeKey"  
 ],  
 "Resource": "arn:aws:kms:example-region:example-account-id-ID:key/example-kms-key-id"  
}

Note: Replace example-region with your AWS Region, example-account-id with your AWS account ID, and example-kms-key-id with your KMS key ID.

Update permissions for managed nodes in a hybrid and multicloud environment

To update permissions for managed nodes in a hybrid and multicloud environment, identify the IAM service role that you used for the hybrid activation. Then, add the CloudWatchAgentServerPolicy permissions policy to the IAM service role.

If you used the default SSMServiceRole for hybrid activation, then complete the following steps:

1. Create an IAM service role, and then attach the AmazonSSMManagedInstanceCore and CloudWatchAgentServerPolicy policies to the role.
2. If you use encrypted log groups, then add the Decrypt, GenerateDataKey, and DescribeKey KMS permissions.
3. Use the service role to create a new hybrid activation.
4. Deregister the existing managed node in the hybrid and multicloud environment.
   Note: When you deregister the managed node, you remove it from Systems Manager.
5. Reregister the managed node with the new activation code and ID.

Check your connection to CloudWatch Logs

Check access to the VPC endpoint for CloudWatch Logs

If your instances are in private subnets without internet access, then SSM Agent must have access to a VPC endpoint for CloudWatch Logs. If you can't access the VPC endpoint for CloudWatch Logs, then you must create one.

To check that your instance can connect to CloudWatch Logs, complete the following steps:

1. Use Session Manager to connect to your instance, and then run the following command:

   nslookup logs.example-region.amazonaws.com

2. Note: Replace example-region with your Region.

3. If the output shows a private IP address and your logs don't appear, then check the security group that's attached to the VPC endpoint. Confirm that the security group allows inbound HTTPS traffic on port 443 from your instance's security group or VPC CIDR range.
   -or-
   If the command fails or resolves to a public IP address, then create a VPC endpoint for CloudWatch Logs.

4. Verify that the security group that's associated with the VPC endpoint allows inbound traffic on port 443 your instance's security group or VPC CIDR range.

Verify that you configured Session Manager preferences to send logs to CloudWatch Logs

Review your Session Manager preferences. Confirm that you selected Enable under CloudWatch logging. Make sure that you specified the correct log group name for CloudWatch log group name. Also, check the log group that you specified for Log group. Then, open the CloudWatch console and verify that the group that you specified appears on the Log groups page.

Identify instance permissions issues

If the preceding troubleshooting steps don't resolve your issue, then complete the following steps:

1. Open the AWS CloudTrail console.
2. Choose Event history.
3. Filter events by the following actions:
   CreateLogStream
   PutLogEvents
   DescribeLogGroups
   DescribeLogStreams

If events don't appear for these actions, then configure SSM Agent to send logs to CloudWatch, and then repeat the preceding steps.

If events appear for the actions that contain error messages, then review the details to identify and resolve missing permissions or configuration issues.