EC2 instance centos7 running on a VPC- Cannot start Amazon Cloudwatch Agent.

Question

Hi there,
I'm trying to install an Amazon cloud watch agent in my instance which running on a VPC ( internal using only, not facing the internet ). to sent Ram usage metric to cloud watch.
I'm following documents on Amazon to create **Interface VPC endpoint**.
1/ Stop Amazon-ssm-agent in my instance.
2/ Install Amazon cloud watch agent and related package.
Manual create config using **amazon-cloudwatch-agent-config-wizard**
change name of config file from config.json to amazon-cloudwatch-agent.json
also add "endpoint_override": "vpc.XXXXXXXXXXXX.logs.ap-southeast-1.vpce.amazonaws.com",

When I'm run manual by command **sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json**

file amazon-cloudwatch-agent.json dissapear in folder etc and new file named file_amazon-cloudwatch-agent.json generated in folder amazon-cloudwatch-agent.d with the same content of amazon-cloudwatch-agent.json.

Checking log file a see the logs like this:
2022-02-09T08:04:01Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: RequestError: send request failed
caused by: Post "https://ec2.ap-southeast-1.amazonaws.com/": dial tcp 52.95.35.53:443: connect: connection timed out

And I can't see the metric in Amazon cloud watch console. Can someone help me to make it work

Edited: after enable debug in log I see more log like this:
2022-02-09T09:53:07Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:07Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:08Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:08Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:09Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:09Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:10Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:10Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:11Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:11Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:12Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:12Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:13Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:13Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:14Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:14Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:15Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:15Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:16Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:16Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics
2022-02-09T09:53:17Z D! [outputs.cloudwatch] Buffer fullness: 0 / 10000 metrics

Answer

Hello Cuong,

I'm understanding your scenario as below, please correct me if I'm wrong.

* Ec2 instance in a private subnet, that does not have access to the Internet. 
* You want to push Memory(Ram usage) metrics to CloudWatch
* You have created VPC Endpoint for CloudWatch Logs

Based on above information, it seems that you are not using the correct VPC Endpoint. To push metrics you need VPC Endpoint for CloudWatch service not CloudWatch Logs, on the contrary you will need VPC Interface Endpoint for CloudWatch Logs to push logs from Private EC2 instance with CW Agent. You can have details from [[1](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Metricssection)] for endpoint_override configuration for Metrics section and [[2](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-and-interface-VPC.html)] for VPC Endpoint for CloudWatch.

In addition to above, when pushing metrics from EC2 instance - EC2 instances needs to connect to EC2 service to obtain the EC2 tags initially. For this purpose you will also need to have VPC Interface endpoint for EC2 service as well. This can be seen from the error that you are observing:

* `2022-02-09T08:04:01Z W! processors.ec2tagger ec2tagger: Unable to describe ec2 tags for initial retrieval: RequestError: send request failed caused by: Post "https://ec2.ap-southeast-1.amazonaws.com/": dial tcp 52.95.35.53:443: connect: connection timed out`

and make sure the EC2 instance's associated IAM Role/Instance Profile has the required permissions as described [[3](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent.html#create-iam-roles-for-cloudwatch-agent-roles)]. Precisely for above error, the required permission is "**ec2:DescribeTags**".

Hope this helps. Looking forward to your feedback.

Thanks

EC2 instance centos7 running on a VPC- Cannot start Amazon Cloudwatch Agent.

Relevant content