New user sign up using AWS Builder ID
New user sign up using AWS Builder ID is currently unavailable on re:Post. To sign up, please use the AWS Management Console instead.
How do I resolve my unified CloudWatch agent not pushing log events?
I want to resolve my unified Amazon CloudWatch agent not pushing log events.
Short description
If your unified CloudWatch agent doesn't push log events, then the following are possible causes:
- Out of sync metadata
- Outdated version of the CloudWatch agent
- Failure to connect to the CloudWatch Logs endpoint
- Incorrect account, AWS Region, or log group configurations
- Insufficient AWS Identity and Access Management (IAM) permissions
- CloudWatch agent run errors
- Timestamp issues
- Missing or incorrect value in the state file for the log
- Incorrect source log file configuration
- PutLogEvents constraints
- High force_flush_interval value
Note: The following resolution doesn't apply to CloudWatch Container Insights. For more information on troubleshooting CloudWatch Container Insights, see Troubleshooting Container Insights and How do I troubleshoot the pod status in Amazon EKS?
Resolution
Out of sync metadata
If you have out of sync metadata, then the CloudWatch agent doesn't work as intended. To resolve this issue, complete the following steps:
- Make sure that you use AWS CloudFormation, AWS Systems Manager Agent (SSM Agent), user data scripts, or the command line to install the CloudWatch agent at launch.
- Make sure that you create an AMI before you install the CloudWatch agent. AMIs capture unique information from the original instance.
- For Windows instances, use Sysprep when you work with an AMI. For more information, see How can I use Sysprep to create and install custom reusable Windows AMIs?
Outdated version of the CloudWatch agent
If you have an outdated version of the CloudWatch agent, then download the CloudWatch agent package release notes and latest version number. The latest version might include updates that resolve your issue.
Failure to connect to the CloudWatch Logs endpoint
To test your connectivity to the CloudWatch Logs endpoint, use one of the following commands:
Note: Replace example-region with the required Region.
telnet logs.<example-region>.amazonaws.com 443
nc -zv logs.<example-region>.amazonaws.com 443
If you receive connection failures, then complete the following steps:
- Make sure that the security group and network access control lists (network ACL) allow connectivity.
- For public endpoints, use an internet gateway or a NAT gateway for your instance.
- If you use Amazon Virtual Private Cloud (Amazon VPC) endpoints, then make sure that the endpoint resolves to an Amazon VPC IP. Also, make sure that the endpoint security group allows access from the source instance.
Incorrect account, Region, or log group configurations
For the CloudWatch agent configuration file, make sure that the specified Region matches the console Region. Also, make sure that the logs are checked in the correct account.
Optionally, use the following common-config.toml files to override system defaults for the CloudWatch agent:
Linux:
/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
/etc/amazon/amazon-cloudwatch-agent/common-config.toml
Windows:
$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml
Insufficient IAM permissions
The CloudWatch agent uses credentials from either the IAM user or IAM role policy to push log events to the CloudWatch service. Before you publish a log event, create a log group and log stream. If there's no log group or log stream, then the CloudWatch agent creates them.
Check that your policy includes the following IAM permissions:
"logs:CreateLogGroup","logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams"
Add any missing IAM permissions to the user policy or the role policy.
Note: When you create IAM roles and users, it's a best practice to use the CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy policies that are created by Amazon.
CloudWatch agent run errors
Verify that the CloudWatch agent is running. If the agent isn't running, then check the log file location for errors:
Linux:
/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
Windows:
$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log
Note: Logs might be specified in a custom logfile location. Check the agent configuration file to identify any custom log locations.
Use the debug parameter to turn on verbose debug logging in the configuration file. If you use the run_as_user parameter, then check that the user has permissions to the log location. If the user doesn't have the necessary permissions, then CloudWatch can't write logs to the location.
If you observe IMDS related issues when you start the CloudWatch agent, then complete the following steps:
- Check if you can access the metadata service. Check the connectivity to the metadata endpoint. For more information, see Access instance metadata for an EC2 instance.
- Review your OS level routes and firewalls to check if there are any rules that block connectivity to the metadata endpoint. For Linux, see How do I troubleshoot instance metadata issues on my EC2 Linux instance? For Windows, see Why does my Amazon EC2 Windows instance generate a "Waiting for the metadata service" error?
Timestamp issues
Check for log event timestamps that are older than 14 days or more than two hours in the future. The PutLogEvents command doesn't allow log batches for either time period. Also, check that the system time service on the instance is correctly configured. For more information, see Change the time zone of your instance.
Missing or incorrect value in the state file for the log
To resolve a missing or incorrect value in the state file for the log, complete the following steps:
-
Make sure that your state files are in the correct location:
Linux:/opt/aws/amazon-cloudwatch-agent/logs/state
Windows:
C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\state
Note: Make sure that the preceding state files contain the same size value as the log file. This allows new data to be read and published to CloudWatch.
-
Check the size value of the state file:
Linux:sudo cat /opt/aws/amazon-cloudwatch-agent/logs/state/_var_log_httpd_access_log
Windows:
gc 'C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\state\C__Program_Files_log.log'
Note: The output of the preceding commands is a numerical value of the current offset. For example, if the output is 34, then the current offset is at the 34th byte of the log file. Logs are read at this byte when new logs are written to the log file.
-
Check the size value of the log file:
Linux:stat /var/log/httpd/access_log | grep Size | awk '{print $2}'
Windows:
Get-Item 'C:\Program Files\log.log' | Format-List | findstr Length
-
Check that the outputs from the preceding step 2 and 3 are the same.
Incorrect source log file configuration
To resolve issues from an incorrect source log file configuration, complete the following steps:
- Check if the CloudWatch agent configuration includes the log file that you want to monitor.
- Make sure that the log file has the correct file format, and make sure that the correct names and locations are used.
- From the current configuration, check that the logs are pushed to the same log group and log stream that you want to retrieve data from.
- Review the log file that you're monitoring on your instance. Check that the source log file contains the data that you want to be published to the log group.
PutLogEvents constraints
CloudWatch agent uses the PutLogEvents API to publish log events to CloudWatch log groups. Make sure that you comply with the constraints of this API. For more information, see PutLogEvents.
High force_flush_interval value
The force_flush_interval specifies the maximum amount of time that the logs can remain in the memory buffer before they are sent to the server. If the size of the logs reaches the maximum buffer of 1 MB, then the logs are immediately published to CloudWatch. The logs are published to CloudWatch regardless of the interval set by the force_flush_interval.
If the interval is set to a high value and the log size doesn't reach 1 MB, then the logs remain in the buffer. The logs remain in the buffer until the buffer is full or the interval elapses. During this interval, the logs aren't available in CloudWatch. Make sure that you set your interval value to the correct size. It's a best practice to set your interval value based on the log size and intervals that you want them to be published at.
Related information

Relevant content
- asked 6 years agolg...
- asked 2 years agolg...
- asked a year agolg...
- Accepted Answerasked 10 months agolg...
- Accepted Answerasked 2 years agolg...
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 8 months ago