Why doesn't the unified CloudWatch agent push my metrics or log events to CloudWatch?

7 minute read

I configured the unified Amazon CloudWatch agent on my Amazon Elastic Compute Cloud (Amazon EC2) instance to post metrics and logs to Amazon CloudWatch. However, I can't see my metrics or log events in the CloudWatch console.

Resolution

Check the agent logs to identify the issue

Before you troubleshoot, check the agent log file to help troubleshoot problems with the unified CloudWatch agent package.

If the agent can't connect to the required AWS service endpoints or Amazon Virtual Private Cloud (Amazon VPC) endpoints, then you might receive the "Agent Log Error: No Connectivity to Endpoint" error.

If you don't have the required permissions to make API calls to CloudWatch, then you receive the "Agent Log Error: Insufficient Permissions" error.

Example "No connectivity" error:

2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout
2021-08-30T04:07:46Z W! 210 retries, going to sleep 1m0s before retrying.
2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout
2021-08-30T04:07:46Z W! 211 retries, going to sleep 1m0s before retrying.

Example "Insufficient permissions" error:

2021-08-30T02:15:45Z E! cloudwatch: code: AccessDenied, message: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData, original error: 
2021-08-30T02:15:45Z W! 1 retries, going to sleep 400ms before retrying.
2021-08-30T02:15:46Z E! WriteToCloudWatch failure, err:  AccessDenied: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData
    status code: 403, request id: f1171fd0-05b6-4f7d-bac2-629c8594c46e

Confirm connectivity to the CloudWatch endpoints

If traffic to CloudWatch goes through Amazon VPC endpoints instead of the public internet, then check the following parameters:

If you use private name servers, then confirm that the DNS resolution returns accurate responses.
Confirm that the CloudWatch endpoints resolve to private IP addresses.
Confirm the security group that's associated with the Amazon VPC endpoint allows inbound traffic from the host.

Then, to confirm connectivity to the CloudWatch endpoints, complete the following steps:

Run the following telnet command to check connectivity to the metrics endpoint:

telnet monitoring.us-east-1.amazonaws.com 443
Trying 52.46.138.115...
Connected to monitoring.amazonaws.com.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Run the following telnet command to check connectivity to the logs endpoint:

telnet logs.us-east-1.amazonaws.com 443
Trying 3.236.94.218...
Connected to logs.us-east-1.amazonaws.com.
Escape character is '^]'.
^]
telnet> quit
Connection closed

Run the following dig command to check that the Amazon VPC endpoint resolves to private IP addresses:

dig monitoring.us-east-1.amazonaws.com 
+short172.31.11.121
172.31.0.13

Review the unified CloudWatch agent configuration

The CloudWatch agent configuration file specifies the metrics and logs that the agent publishes to CloudWatch. Review the agent configuration file to confirm that the file includes the logs and metrics you want to publish.

Confirm that the host has permissions to publish metrics and logs

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

The AWS managed policies CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy define the permissions required to deploy the unified CloudWatch agent. Use these policies as a reference to confirm that your host has the correct permissions.

The following AWS CLI config command shows that the Amazon EC2 instance is missing the AWS Identity and Access Management (IAM) role:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Region: us-east-1
credsConfig: map[]
Error in retrieving parameter store content: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Fail to fetch/remove json config: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Fail to fetch the config!

The following AWS CLI config command shows that the EC2 instance has the incorrect IAM role:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Region: us-east-1
credsConfig: map[]
Error in retrieving parameter store content: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server
    status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4d
Fail to fetch/remove json config: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server
    status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4d

Fail to fetch the config!

The following get-caller-identity command returns the IAM user or role that's associated with the instance:

aws sts get-caller-identity
{
    "UserId": "AROA123456789012ABCDE:i-0744de7c842d2c2ba",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/CloudWatchAgentServerRole/i-0744de7c842d2c2ba"
}

Confirm that the agent correctly starts

To start the agent, run the following valid start commands.

Linux commands

If the configuration file is stored locally on the instance, then run the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path

If the configuration file is stored in Parameter Store, a capability of AWS Systems Manager, then run the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name

Windows commands

If the configuration file is stored locally on the instance, then run the following command:

& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json

If the configuration file is stored in Parameter Store, then run the following command:

& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name

Important: Don't start the agent from the Windows Control Panel.

Confirm that the agent runs

To publish metrics and logs, the agent must be active. To verify that the CloudWatch agent is active, run the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
{
    "status": "running",
    "starttime": "2021-08-30T02:13:44+00:00",
    "configstatus": "configured",
    "cwoc_status": "stopped",
    "cwoc_starttime": "",
    "cwoc_configstatus": "not configured",
    "version": "1.247349.0b251399"
}

Restart the agent after you update the agent configuration

The agent doesn't automatically register changes to the configuration file. If you update the agent configuration to include new or different metrics and logs, then run the following command to restart the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop
****** processing cwagent-otel-collector ******
cwagent-otel-collector has already been stopped

****** processing amazon-cloudwatch-agent ******
Redirecting to /bin/systemctl stop amazon-cloudwatch-agent.service

Run the following command to fetch the updated configuration file and restart the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
2021/08/31 02:45:37 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...
Valid Json input schema.
I! Detecting run_as_user...
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
amazon-cloudwatch-agent has already been stopped
Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service

Note: Replace config.json with the path to your configuration file.

Run the following command to verify that the agent is running:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
{
  "status": "running",
  "starttime": "2021-08-31T02:45:37+0000",
  "configstatus": "configured",
  "cwoc_status": "stopped",
  "cwoc_starttime": "",
  "cwoc_configstatus": "not configured",
  "version": "1.247349.0b251399"
}

In the output, confirm that the status shows running. When the status is running, this confirms that the CloudWatch agent is in a running state.

Related information

How do I install and configure the unified CloudWatch agent to send metrics and logs from my Amazon EC2 instance to CloudWatch?

Topics: Management & Governance
Tags: Amazon CloudWatch
Language: English

AWS OFFICIALUpdated 5 months ago

1 Comment

This article was reviewed and updated on 2026-03-23.

MODERATOR

AWS Official

replied 4 months ago

Relevant content

Can I use Cloudwatch Unified Agent to collect user logs from my application's client interface?
Accepted Answer
IndieGameDeveloperFromParallel...
asked 2 years ago
Cloudwatch agent don't send all basic metrics or logs
Renan
asked 3 years ago
Why is CloudWatch Agent not collecting JMX Metrics?
Stein
asked a year ago
how to push only error and warning logs to cloudwatch
AWS-User-2503537
asked 4 years ago
My EC2 instance is in Commercial Regions. How should I configure the CloudWatch agent to collect metrics in the local region and upload logs to AWS GovCloud Regions?
rePost-User-6814093
asked 2 years ago
Why doesn't my unified CloudWatch agent send log events?
AWS OFFICIALUpdated 7 months ago
How do I troubleshoot high CPUUtilization from the unified CloudWatch agent on my server?
AWS OFFICIALUpdated 3 years ago
How do I troubleshoot an alarm that monitors CloudWatch agent metrics and is in the INSUFFICIENT_DATA state?
AWS OFFICIALUpdated 8 months ago
Why can’t I push log data to CloudWatch Logs with the awslogs agent?
AWS OFFICIALUpdated 4 years ago
Logging EC2 Instances Bash Commands with CloudWatch Logs
EXPERT
AWS-Wole
published a year ago