Why isn't the unified CloudWatch agent pushing log events?

Short description

The following issues can prevent the unified CloudWatch agent from pushing log events:

• Out-of-sync metadata caused by creating an Amazon Machine Image (AMI) after the CloudWatch agent is installed
• Using an outdated version of the CloudWatch agent
• Failure to connect to the CloudWatch Logs endpoint
• Incorrect account, Region, or log group configurations
• Insufficient AWS Identity and Access Management (IAM) permissions
• CloudWatch agent run errors
• Timestamp issues

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

Important: The following troubleshooting steps don't apply to Container Insights. For details on troubleshooting Container Insights, see Troubleshooting Container Insights and How can I troubleshoot pod status in Amazon EKS?

Review the installation method for the CloudWatch agent

It's a best practice to install the CloudWatch agent at launch using AWS CloudFormation, AWS Systems Manager Agent (SSM Agent), user data scripts, or the AWS CLI. It is also a best practice to create an AMI before installing the CloudWatch agent. AMIs typically capture unique information from the original instance. Metadata becomes out of sync, and this state can lead to the CloudWatch agent not working as intended. Out-of-sync metadata is the reason that many Windows instances require Sysprep when working with AMI. For more information, see How can I use Sysprep to create and install custom reusable Windows AMIs in Amazon EC2?

Confirm that you're using the latest version of the CloudWatch agent

Download and review the README files for the CloudWatch agent release notes and latest version number. If you're using an older version of the CloudWatch agent, be sure to upgrade. The latest version might include updates that resolve the issue that you're experiencing.

Test connectivity to your CloudWatch Logs endpoint

Test connectivity to the CloudWatch Logs endpoint using either of the following commands:

telnet logs.<region>.amazonaws.com 443

nc -zv logs.<region>.amazonaws.com 443

If you get connectivity failures, be sure that:

• The security group and network access control list (ACL) rules allow connectivity
• Your instance can reach the public endpoint using an internet gateway or a network address translation (NAT) gateway
• If you're using VPC endpoints, the endpoint resolves to a VPC IP and the endpoint security group allows access from the source instance

Review your account, Region, and log group configurations

In the CloudWatch agent configuration file:

• Be sure that the specified Region matches the console Region
• Verify that logs are checked in the correct account

Optionally, you can use the common-config.toml file to override system defaults for the CloudWatch agent. These system defaults include the proxy, Region, and credential information for the agent. The file is available in the following locations.

Linux:

/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

-or-

/etc/amazon/amazon-cloudwatch-agent/common-config.toml

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml

Check your IAM permissions

The CloudWatch agent uses credentials from either the IAM user or IAM role policy to push log events to the CloudWatch service. Before a log event can be published, you must create a log group and log stream. If there's no log group or log stream, the CloudWatch agent creates them.

Confirm that your policy includes the following IAM permissions:

"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"

Add any missing IAM permissions to the user policy or the role policy.

Note: When creating IAM roles and users, it's a best practice to use the CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy policies created by Amazon rather than custom policies.

Resolve CloudWatch agent run errors

Verify that the CloudWatch agent is running. If the agent isn't running, check the log file for errors and resolve them. Log files are located in the following locations.

Linux:

/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log

Note: Logs might be specified in a custom logfile location. Check the agent configuration file to identify any custom log locations.

In the agent configuration file, enable verbose debug logging using the debug parameter. If you're using the run_as_user parameter, confirm that the user has permissions to the log location path. Without the necessary permissions, the CloudWatch agent can't write logs to the location.

Resolve timestamp issues

Check for log event timestamps that are older than 14 days or more than two hours in the future. The PutLogEvents command doesn't allow log batches in either time frame.

Also, verify that the system time service on the instance is correctly configured. For more information, see Set the time for your Linux instance or Set the time for a Windows instance.

---

Related information

Troubleshooting the CloudWatch agent