The intention of this documentation is to provide the building blocks to create critical CloudWatch alarms which are fit for onboarding to Incident Detection and Response. It contains specific alarm best practices for AWS Services.
Overview
Here we have included Alarm Best Practices for Elastic Kubernetes Service that Incident Detection and Response (IDR) customers can refer to. They can use these best practices as a starting point in creating alarms fit to be onboarded to IDR. There are also AWS recommended alarms available, which customers may make use of.
The CloudWatch alarm best practices provided on this page are intended solely as general guidelines to assist you in configuring alarms for the Incident Detection and Response service. The final selection, configuration, and implementation of CloudWatch alarms should be tailored to the unique requirements of your environment, workloads, and operational needs. The recommendations we provide do not guarantee comprehensive coverage or detection of all issues, incidents, or outages. For more information on detailed configuration, we recommend consulting with your technical team.
Recommended Metrics to Monitor: EKS Service - Option 1 - Install Container Insights (AWS Service)
Prerequisite: Setup Container Insights on Amazon EKS and Kubernetes. For best results, use the latest version of Container Insights with Enhanced Observability. This enhanced version works only with EC2 instances, Fargate nodes are not supported.
CloudWatch Container Insights provides a comprehensive metrics monitoring solution for Amazon EKS at the cluster, node, pod, task, and service level.

Recommended Metrics to Monitor: EKS Service - Option 1 Enhanced - ContainerInsights together with Prometheus Integration
Prerequisite: Leverage CloudWatch Container Insights Prometheus Support to ingest the Prometheus metrics into CloudWatch.
Make application and business level metrics visible in CloudWatch directly, and monitor via CloudWatch Alarm or Dashboard.

Recommended Metrics to Monitor: EKS Service - Option 2 - Install Prometheus (Open Source or AWS Managed)
Prerequisite: Amazon EKS supports multiple options for metrics capture with Prometheus. The Amazon EKS control plane provides a metrics endpoint that exposes metrics in a Prometheus format. You can deploy Prometheus into your Amazon EKS cluster to consume these metrics.
Kubernetes exposes control plane metrics in a Prometheus format by using the /metrics HTTP API endpoint. You should install Prometheus in your Kubernetes cluster to graph and view these metrics with a web browser. You can also ingest the metrics exposed by the Kubernetes API server into CloudWatch.
This deployment is considered as a 3rd Party APM, which needs to integrate with IDR tools via Webhook or SNS protocol.

Recommended Metrics to Monitor: Customer Managed Kubernetes
Prerequisite: Open Source installation. Not an AWS service. This repository collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
This deployment is considered as a 3rd Party APM, which needs to integrate with IDR tools via Webhook or SNS protocol.

Additional Resources
For additional information on Cloudwatch metrics and dimensions for EKS please refer to the following documentation: EKS Container Insights CloudWatch Metrics
For additional information on EKS Kubernetes control plane metrics exposed in a Prometheus format please see the following documentation: EKS Kubernetes control plane metrics exposed in a Prometheus format (3rd party link)