Why isn't the unified CloudWatch agent pushing my metrics or log events to CloudWatch?
I have configured the unified CloudWatch agent on my Amazon Elastic Compute Cloud (Amazon EC2) instance to post metrics and logs to Amazon CloudWatch. But I can't see my metrics or logs in the CloudWatch console. Why isn't the unified CloudWatch agent pushing my metrics and logs to CloudWatch?
Short description
There are a number of reasons why the unified CloudWatch agent might not be pushing your metrics or logs to CloudWatch. For example, you might have a permissions or connectivity error that is preventing the agent from publishing your metrics. When you review the unified CloudWatch agent logs, you might see an error like this:
- Agent Log Error: No Connectivity to Endpoint
- Agent Log Error: Insufficient Permissions
Resolution
Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.
Review the unified CloudWatch agent logs
Use the agent log file to help you troubleshoot problems you encounter with the unified CloudWatch agent package. You might be experiencing one of these common issues:
- You're experiencing connectivity issues to the required AWS service endpoints or VPC endpoints.
- You don't have the correct permissions to make support API calls to CloudWatch.
- The log file isn't present on your local file system.
You might see one of these errors in the logs:
Agent Log Error: No Connectivity to Endpoint
2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout 2021-08-30T04:07:46Z W! 210 retries, going to sleep 1m0s before retrying. 2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout 2021-08-30T04:07:46Z W! 211 retries, going to sleep 1m0s before retrying.
Agent Log Error: Insufficient Permissions
2021-08-30T02:15:45Z E! cloudwatch: code: AccessDenied, message: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData, original error: 2021-08-30T02:15:45Z W! 1 retries, going to sleep 400ms before retrying. 2021-08-30T02:15:46Z E! WriteToCloudWatch failure, err: AccessDenied: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData status code: 403, request id: f1171fd0-05b6-4f7d-bac2-629c8594c46e
Confirm connectivity to the CloudWatch endpoints
When traffic to CloudWatch should not transit the public internet, you can use VPC endpoints instead. If you are using VPC endpoints, check the following:
- If you are using private nameservers, confirm that DNS resolution provided accurate responses.
- Confirm that the CloudWatch endpoints resolve to private IP addresses.
- Confirm the security group associated with the VPC endpoint allows inbound traffic from the host.
1. Check connectivity to the metrics endpoint:
$ telnet monitoring.us-east-1.amazonaws.com 443 Trying 52.46.138.115... Connected to monitoring.amazonaws.com. Escape character is '^]'. ^] telnet> quit Connection closed.
2. Check connectivity to the logs endpoint:
$ telnet logs.us-east-1.amazonaws.com 443 Trying 3.236.94.218... Connected to logs.us-east-1.amazonaws.com. Escape character is '^]'. ^] telnet> quit Connection closed
3. Check that the VPC endpoint resolves to private IP addresses:
$ dig monitoring.us-east-1.amazonaws.com +short 172.31.11.121 172.31.0.13
Review the unified CloudWatch agent configuration
The agent configuration file details the metrics and logs that are published to CloudWatch. Review the agent configuration file to confirm that the logs and metrics you want to publish are included.
Confirm that the host has permissions to publish metrics and logs
The AWS managed policies CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy can help you deploy the unified CloudWatch agent and check that you have the correct permissions. Use these policies as a reference to make sure that your host has the correct permissions.
The AWS CLI outputs in these examples show insufficient permissions.
This agent start command output shows that there is no IAM role attached to EC2 instance:
$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s ****** processing amazon-cloudwatch-agent ****** /opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default Region: us-east-1 credsConfig: map[] Error in retrieving parameter store content: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors Fail to fetch/remove json config: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors Fail to fetch the config!
This agent start command output shows that the incorrect IAM role is attached to the EC2 instance:
$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s ****** processing amazon-cloudwatch-agent ****** /opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default Region: us-east-1 credsConfig: map[] Error in retrieving parameter store content: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4d Fail to fetch/remove json config: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4d Fail to fetch the config!
In some cases, an IAM user might be at the command line. The obtain user/role command returns the IAM user or role associated with the instance:
$ aws sts get-caller-identity { "UserId": "AROA123456789012ABCDE:i-0744de7c842d2c2ba", "Account": "123456789012", "Arn": "arn:aws:sts::123456789012:assumed-role/CloudWatchAgentServerRole/i-0744de7c842d2c2ba" }
Confirm that the agent starts correctly
The agent is designed to be started using the AWS CLI with the configuration file passed as an argument. Use these valid start commands.
Linux Commands:
- `$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path` - `$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name`
Windows Commands:
- `& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json"` - `& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name`
Important: Don't start the agent from the Windows Control Panel.
Confirm that the agent is running
To publish metrics and logs, the agent must be running. Run this command to confirm that the agent is active.
$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status { "status": "running", "starttime": "2021-08-30T02:13:44+00:00", "configstatus": "configured", "cwoc_status": "stopped", "cwoc_starttime": "", "cwoc_configstatus": "not configured", "version": "1.247349.0b251399" }
Restart the agent after updating the agent configuration
The agent doesn't automatically register changes to the configuration file. If the agent configuration is updated to include new or different metrics and logs, then restart the agent using this command:
$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop ****** processing cwagent-otel-collector ****** cwagent-otel-collector has already been stopped ****** processing amazon-cloudwatch-agent ****** Redirecting to /bin/systemctl stop amazon-cloudwatch-agent.service $ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:config.json ****** processing amazon-cloudwatch-agent ****** /opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp Start configuration validation... /opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default 2021/08/31 02:45:37 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ... Valid Json input schema. I! Detecting run_as_user... Configuration validation first phase succeeded /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml Configuration validation second phase succeeded Configuration validation succeeded amazon-cloudwatch-agent has already been stopped Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service $ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status { "status": "running", "starttime": "2021-08-31T02:45:37+0000", "configstatus": "configured", "cwoc_status": "stopped", "cwoc_starttime": "", "cwoc_configstatus": "not configured", "version": "1.247349.0b251399" }
Related information

Relevant content
- Accepted Answerasked 10 months agolg...
- asked 10 months agolg...
- asked 6 months agolg...
- asked a year agolg...
- asked 4 years agolg...
- AWS OFFICIALUpdated 10 months ago
- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated 5 months ago
- AWS OFFICIALUpdated 5 months ago