How to setup AWS CloudWatch Container insights to collect only custom EKS metrics (node and pod metrics)?

Question

AWS CloudWatch is a great monitoring and logging solution. CloudWatch dashboards, Metrics, Logs all offer a centralized platform for infrastructure monitoring and application logging. AWS CloudWatch Container insights is one other add on especially for teams using AWS Elastic Kubernetes Service (EKS). As per the documentation, once container insights is setup, it collects all metrics and as per pricing you have to pay for each metric. As see here on the pricing page examples - https://aws.amazon.com/cloudwatch/pricing/ this cost could shot way up yet one would only be interested in collecting node metrics and pod metrics (cpu utilization, memory usage, disk utilization). 
How can one go about configuring container insights to collect custom metrics (filter out metrics they are interested in)?

Answer

Currently there is no native solution from CloudWatch side to limit the metrics for Container Insights. AWS Distro for OpenTelemetry (ADOT) is an AWS-provided distribution of the OpenTelemetry project. The ADOT Collector receives and exports data from multiple sources and destinations. This will enable you to perform advanced configurations, such as customizing metrics that are sent to CloudWatch.

Reference [1]

https://aws.amazon.com/blogs/containers/cost-savings-by-customizing-metrics-sent-by-container-insights-in-amazon-eks/  
Reference[2]

https://aws.amazon.com/blogs/containers/introducing-amazon-cloudwatch-container-insights-for-amazon-eks-fargate-using-aws-distro-for-opentelemetry/

How to setup AWS CloudWatch Container insights to collect only custom EKS metrics (node and pod metrics)?

Relevant content