Why doesn't the unified CloudWatch agent push my metrics or log events to CloudWatch?

7 minute read
2

I configured the unified Amazon CloudWatch agent on my Amazon Elastic Compute Cloud (Amazon EC2) instance to post metrics and logs to Amazon CloudWatch. But I can't see my metrics or logs in the CloudWatch console. I want to see my metrics and log events in CloudWatch

Short description

There might be several reasons why the unified CloudWatch agent doesn't push your metrics or logs to CloudWatch. For example, you might have a permissions or connectivity error that prevents the agent from publishing your metrics. When you review the unified CloudWatch agent logs, one of the following errors might appear:

  • Agent Log Error: No Connectivity to Endpoint
  • Agent Log Error: Insufficient Permissions

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you use the most recent AWS CLI version.

Review the unified CloudWatch agent logs

You can use the agent log file to help troubleshoot problems that you encounter with the unified CloudWatch agent package.

You might experience one the following issues:

You might see one of these errors in the following logs.

Agent Log Error: No Connectivity to Endpoint

2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout
2021-08-30T04:07:46Z W! 210 retries, going to sleep 1m0s before retrying.
2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout
2021-08-30T04:07:46Z W! 211 retries, going to sleep 1m0s before retrying.

Agent Log Error: Insufficient Permissions

2021-08-30T02:15:45Z E! cloudwatch: code: AccessDenied, message: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData, original error: 
2021-08-30T02:15:45Z W! 1 retries, going to sleep 400ms before retrying.
2021-08-30T02:15:46Z E! WriteToCloudWatch failure, err:  AccessDenied: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData
    status code: 403, request id: f1171fd0-05b6-4f7d-bac2-629c8594c46e

Confirm connectivity to the CloudWatch endpoints

If traffic to CloudWatch doesn't go through the public internet, then you can use Amazon VPC endpoints. If you use Amazon VPC endpoints, then check the following parameters:

  • If you use private nameservers, then confirm that DNS resolution provided accurate responses.
  • Confirm that the CloudWatch endpoints resolve to private IP addresses.
  • Confirm the security group associated with the Amazon VPC endpoint that allows inbound traffic from the host.

To confirm connectivity to the CloudWatch endpoints, complete the following steps:

  1. To check connectivity to the metrics endpoint, run the following command:

    $ telnet monitoring.us-east-1.amazonaws.com 443
    Trying 52.46.138.115...
    Connected to monitoring.amazonaws.com.
    Escape character is '^]'.
    ^]
    telnet> quit
    Connection closed.
  2. To check connectivity to the logs endpoint, run the following command:

    $ telnet logs.us-east-1.amazonaws.com 443
    Trying 3.236.94.218...
    Connected to logs.us-east-1.amazonaws.com.
    Escape character is '^]'.
    ^]
    telnet> quit
    Connection closed
  3. To check that the Amazon VPC endpoint resolves to private IP addresses, run the following command:

    $ dig monitoring.us-east-1.amazonaws.com 
    +short172.31.11.121
    172.31.0.13

Review the unified CloudWatch agent configuration

The agent configuration file details the metrics and logs that are published to CloudWatch. Review the agent configuration file to confirm that the logs and metrics that you want to publish are included.

Confirm that the host has permissions to publish metrics and logs

The AWS managed policies CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy can help you deploy the unified CloudWatch agent. The policies can also help check that you have the correct permissions. Use these policies as a reference to make sure that your host has the correct permissions.

The AWS CLI outputs in these examples show insufficient permissions.

The following AWS CLI config command shows that the AWS Identity and Access Management (IAM) role attached to EC2 instance is missing:

$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Region: us-east-1
credsConfig: map[]
Error in retrieving parameter store content: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Fail to fetch/remove json config: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Fail to fetch the config!

The following AWS CLI config command shows that the incorrect IAM role is attached to the EC2 instance:

$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Region: us-east-1
credsConfig: map[]
Error in retrieving parameter store content: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server
    status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4d
Fail to fetch/remove json config: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server
    status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4d

Fail to fetch the config!

The following get-caller-identity command returns the IAM user or role that's associated with the instance:

$ aws sts get-caller-identity
{
    "UserId": "AROA123456789012ABCDE:i-0744de7c842d2c2ba",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/CloudWatchAgentServerRole/i-0744de7c842d2c2ba"
}

Confirm that the agent correctly starts

You can use AWS CLI with the configuration file passed as an argument to start the agent. To start the agent, run the following valid start commands.

For Linux, run the following commands:

- `$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path`
- `$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name`

For Windows, run the following commands:

- `& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json"`
- `& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name`

Important: Don't start the agent from the Windows Control Panel.

Confirm that the agent runs

To publish metrics and logs, the agent must be active. To confirm that the agent is active, run the following command:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
{
    "status": "running",
    "starttime": "2021-08-30T02:13:44+00:00",
    "configstatus": "configured",
    "cwoc_status": "stopped",
    "cwoc_starttime": "",
    "cwoc_configstatus": "not configured",
    "version": "1.247349.0b251399"
}

Restart the agent after you update the agent configuration

The agent doesn't automatically register changes to the configuration file. If the agent configuration is updated to include new or different metrics and logs, then you must use the following command to restart the agent:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop
****** processing cwagent-otel-collector ******
cwagent-otel-collector has already been stopped

****** processing amazon-cloudwatch-agent ******
Redirecting to /bin/systemctl stop amazon-cloudwatch-agent.service


$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:config.json
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
2021/08/31 02:45:37 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...
Valid Json input schema.
I! Detecting run_as_user...
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
amazon-cloudwatch-agent has already been stopped
Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
{
  "status": "running",
  "starttime": "2021-08-31T02:45:37+0000",
  "configstatus": "configured",
  "cwoc_status": "stopped",
  "cwoc_starttime": "",
  "cwoc_configstatus": "not configured",
  "version": "1.247349.0b251399"
}

Related information

How do I install and configure the unified CloudWatch agent to push metrics and logs from my EC2 instance to CloudWatch?

AWS OFFICIAL
AWS OFFICIALUpdated 5 days ago