Why doesn't my unified CloudWatch agent that send log events?

Resolution

The following resolution doesn't apply to CloudWatch Container Insights. To troubleshoot CloudWatch Container Insights, see Troubleshooting Container Insights.

Out of sync metadata

Metadata that's out of sync can cause issues with your CloudWatch agent.

To resolve this issue, take the following actions:

• Before you install the CloudWatch agent, create an Amazon Machine Image (AMI) to capture unique information from the original instance.
• Use AWS CloudFormation, AWS Systems Manager, user data scripts, or the command line to install the CloudWatch agent at launch.
• For Windows instances, use Sysprep when you work with an AMI.

Outdated version of the CloudWatch agent

Check the version of the CloudWatch agent that you're currently using. If the CloudWatch agent is outdated, then download and install the CloudWatch agent package with the latest version. The latest version might include updates that resolve your issue.

Failure to connect to the CloudWatch Logs endpoint

Note: In the following commands, replace example-region with your AWS Region.

To test your connection to the CloudWatch Logs endpoint, run one of the following commands:

telnet logs.example-region.amazonaws.com 443

nc -zv logs.example-region.amazonaws.com 443

By default, netcat isn't installed for Windows machines. Instead, run the following Test-NetConnection command on Windows PowerShell:

Test-NetConnection -ComputerName logs.example-region.amazonaws.com -Port 443

If the connection fails, then take the following actions:

• Make sure that the security group and network access control lists (network ACLs) allow the connection.
• For public endpoints, use an internet gateway or a NAT gateway for your instance.
• If you use Amazon Virtual Private Cloud (Amazon VPC) endpoints, then make sure that the endpoint resolves to an Amazon VPC IP address. Also, make sure that the endpoint security group allows access from the source instance.

Incorrect account, Region, or log group configurations

For the CloudWatch agent configuration file, make sure that the specified Region matches the console Region. Also, make sure to check the logs in the correct AWS account.

To override system defaults for the CloudWatch agent, use the following common-config.toml file locations.

Linux:

/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

/etc/amazon/amazon-cloudwatch-agent/common-config.toml

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml

Insufficient IAM permissions

To send log events to CloudWatch, the CloudWatch agent uses credentials from either the AWS Identity and Access Management (IAM) user or IAM role policy. 

Check that your identity-based policy includes the following IAM permissions:

• logs:CreateLogGroup
• logs:CreateLogStream
• logs:PutLogEvents
• logs:DescribeLogStreams

Add missing IAM permissions to the user policy or the role policy.

When you create the IAM role or user, it's a best practice to use the CloudWatchAgentServerPolicy or CloudWatchAgentAdminPolicy policy.

If there's no log group or log stream, then the CloudWatch agent creates them. Before you publish a log event, create a log group and log stream.

CloudWatch agent run errors

Verify that the CloudWatch agent is running. If the agent isn't running, then check the log file location for errors.

For Linux, the log file location is /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log.

For Windows, the log file location is $Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log.

You might have specified logs in a custom logfile location. To identify custom log locations, check the agent configuration file.

Use the debug parameter to turn on verbose debug logging in the configuration file. If you use the run_as_user parameter, then check that the user has permissions to the log location. If the user doesn't have the necessary permissions, then CloudWatch can't write logs to the location.

If you experience issues with the international material data system (IMDS) when you start the CloudWatch agent, then take the following actions:

• Confirm that you can access the metadata for your Amazon Elastic Compute Cloud (Amazon EC2) instance. Check the connection to the metadata endpoint.
• In your operating system (OS)-level routes and firewalls, check whether you created rules that block connectivity to the metadata endpoint. For Linux, see How do I troubleshoot instance metadata issues on my EC2 Linux instance? For Windows, see How do I troubleshoot the "Waiting for the metadata service" error on my Amazon EC2 Windows instance?

Timestamp issues

The CloudWatch agent uses the PutLogEvents API operation to publish log events to CloudWatch log groups. Check that your log events meet the PutLogEvents API operation constraints. Also, check that you correctly configured the time zone of your instance.

Incorrect value in the state file for the log

To resolve a missing or incorrect value in the state file for the log, complete the following steps:

1. Confirm that your state files are in the correct location.
   For Linux, use the following file location:
   /opt/aws/amazon-cloudwatch-agent/logs/state
   For Windows, use the following file location:
   C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\state
   Note: Make sure that the preceding state files contain the same size value as the log file so that the CloudWatch agent can read and publish new data to CloudWatch log groups.

2. Check the size value of the state file.
   For Linux, run the following sudo command:

   sudo cat /opt/aws/amazon-cloudwatch-agent/logs/state/_var_log_httpd_access_log

   For Windows, run the following Windows PowerShell command:

   gc 'C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\state\C__Program_Files_log.log'

   Note: The output of the preceding commands contains a numerical value of the current offset. For example, if the output is 34, then the current offset is at the 34th byte of the log file. CloudWatch Logs read at the identified byte when the CloudWatch agent writes new logs to the log file.

3. Check the size value of the log file.
   For Linux, run the following command:

   stat /var/log/httpd/access_log | grep Size | awk '{print $2}'

   For Windows, run the following command:

   Get-Item 'C:\Program Files\log.log' | Format-List | findstr Length

4. Check that you get the same outputs from steps 2 and 3.

Incorrect source log file configuration

To resolve issues because of an incorrect source log file configuration, take the following actions:

• Check whether the CloudWatch agent configuration includes the log file that you want to monitor.
• Make sure that the log file has the correct file format, and you used the correct names and locations.
• In the current configuration, check that you configured the logs to be sent to the same log group and log stream that you want to retrieve data from.
• Review the log file that you're monitoring on your instance. Check that the source log file contains the data that you want to publish to the log group.

High force_flush_interval value

If the log size reaches the maximum buffer of 1 MB, then CloudWatch immediately receives the logs regardless of the force_flush_interval interval that you set. For more information, select the CloudWatch agent configuration file: Metrics section tab.

If you set the interval to a high value and the log size doesn't reach 1 MB, then the logs remain in the buffer. The CloudWatch agent sends logs to the server only after the buffer is full or the interval elapses. During this interval, the logs aren't available in CloudWatch. Make sure that you set your interval value to the correct size. It's a best practice to set your interval value based on the log size and intervals that you want to publish them.

Related information

Troubleshooting the CloudWatch agent

How do I troubleshoot the pod status in Amazon Elastic Kubernetes Service (Amazon EKS)?