ec2tagger: Unable to describe ec2 tags for initial retrieval: AuthFailure: AWS was not able to validate the provided access credentials / cloudwatch log agent, vpc endpoints

Question

I got error: "ec2tagger: Unable to describe ec2 tags for initial retrieval: AuthFailure: AWS was not able to validate the provided access credentials" in cloudwatch log agent on an ec2 instance that has:

1. CloudWatchAgentServerRole -- this is default AWS managed role attached to the instance, this default role already allow ""ec2:DescribeTags"," in its policy. <---- NOTE this
2. Its NACL allowed all outbound and allowed all vpc's CIDR network range inbound
3. Cloudwatch log agent config file's region is correct
4. telnet ec2.us-east-2.amazonaws.com 443 or telnet monitoring.us-east-2.amazonaws.com 443 or telnet logs.us-east-2.amazonaws.com 443 under the ec2 instance all return successful connection (Connected <..> Escape character is '^]')

I also create three vpc endpoints: logs (com.amazonaws.us-east-2.logs), monitoring (com.amazonaws.us-east-2.monitoring), ec2 (com.amazonaws.us-east-2.ec2) interface endpoints. They have SG that allowed all VPC's CIDR network range inbound.

The idea is to expose metrics to cloudwatch via vpc endpoints.

Despite all above setup, I can't make cloudwatch agent to work and it keeps echo above error complain about credentials is not valid even though the REGION in config file is correct and traffic between instance and cloudwatch is allowed.

Answer

There is no need to explicitly specify the region in the CloudWatch agent configuration for an EC2 instance. The agent will automatically send the logs to the region where the instance is located. You only need to specify the region if you want to send your logs to a different region. See [the documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html) for details.

This error (**AWS was not able to validate the provided access credentials**) occurs when the credentials the API calls are signed with are either invalid or have invalid timestamp. There might be multiples causes for this issue:

- Ensure you have **no** configured AWS credentials on the EC2 instance (there should be no credentials in `~/.aws/config` or `~/.aws/credentials` or in the environment variables or in the agent configuration file) - the credentials will be obtained automatically by the EC2 instance using the STS service
- Make sure the system time on your EC2 instance is valid and synchronized (including the time zone)
- If you are running your instances in one of the new regions not enabled by default (such as **Asia Pacific (Hong Kong)**), consider configuring the STS global endpoint to issue tokens using the new token format, see this [knowledge center post](https://aws.amazon.com/premiumsupport/knowledge-center/iam-validate-access-credentials/) for details

ec2tagger: Unable to describe ec2 tags for initial retrieval: AuthFailure: AWS was not able to validate the provided access credentials / cloudwatch log agent, vpc endpoints

Relevant content