How do I install Kubernetes Metrics Server in my Amazon EKS clusters and troubleshoot issues?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Verify whether Metrics Server is installed and activated. Run the following commands:

kubectl get APIService | grep metrics-server

kubectl get deployment metrics-server -n kube-system

kubectl get pods -A | grep metrics-server

kubectl get apiservice v1beta1.metrics.k8s.io -o json | jq '.status'

If the metrics-server is active, then you see an entry for v1beta1.metrics.k8s.io in the output for returned services. If you don't see the correct entry, then install Metrics Server.

Install Metrics Server

1. Install Metrics Server as a community add-on or Amazon EKS API.
   Note: If you use an AWS Fargate launch type, then update the metrics-server deployment configuration. To do this, update containerPort from 10250 to 10251, and then update --secure-port=10250 to --secure-port=10251. You can select any available port number within the 1024-65535 range. Make sure that your security groups allow access to the new port from the control plane security group to worker node security.
2. Verify that the metrics-server deployment runs the desired number of pods. Run the following command:

   kubectl get deployment metrics-server -n kube-system

Troubleshoot Metrics Server issues

To check that Metrics Server works, display CPU or memory resource usage of nodes or pods. Run the following commands:

Nodes:

kubectl top nodes

Pods:

kubectl top pods

If you receive an error, then complete the steps in one of the following sections based on the error message that you receive.

Error from server (Forbidden)

If you receive the error message, "Error from server (Forbidden)", then update your Kubernetes Role-Based Access Control (RBAC) configuration. The Kubernetes RBAC identity needs sufficient permissions to read cluster metrics. For more information, see the following resources:

• rbac.yaml on the GitHub website
• Associate IAM identities with Kubernetes permissions
• Using RBAC authorization on the Kubernetes website

Complete the following steps:

1. If your Amazon EKS clusters authenticate with the Amazon EKS API, then check whether there's an access entry for the AWS Identity and Access Management (IAM) identity. Run the list-access-entries command:

   aws eks list-access-entries --cluster-name your_cluster_name

   Note: Replace your_cluster_name with the name of your cluster. If there isn't an access entry for the IAM identity, then create an access entry.

2. If you access your cluster through a role that's defined in the aws-auth ConfigMap, then confirm that you set the username field and mapping. To describe the aws-auth ConfigMap, run the following command:

   $ kubectl describe -n kube-system configmap aws-auth

3. If the IAM identity isn't listed or isn't correctly configured, then update the IAM principals of the configuration map. For more information, see Add IAM principals to your Amazon EKS cluster.

4. You might not receive metrics even when the Metrics Server API service v1beta1.metrics.k8s.io is set to True. Confirm that ClusterRole and ClusterRoleBindings have the required permissions. For more information, see rbac.yaml on the GitHub website.

Unable to authenticate the request due to an error: x509: certificate signed by unknown authority

If Metrics Server reports that it can't authenticate the client certificate, then you might receive an error message.

To check the logs, run the following command:

kubectl logs -n kube-system -l k8s-app=metrics-server —container metrics-server

If there's a problem with a front-proxy certificate, then the logs might contain a message similar to the following:

E0413 12:28:25.449973 1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority

The kubelet certificate must be signed by the cluster Certificate Authority (CA). To fix the error and turn off certificate validation, edit the metric server deployment in the kube-system namespace. To do this, add ––kubelet-insecure-tls under the container spec. Run the following command:

kubectl edit deployment metrics-server -n kube-system
--
Image:           602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/metrics-server:v0.7.2-eksbuild.1
    Port:            10250/TCP
    Host Port:       0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      --secure-port=10250 
      --cert-dir=/tmp
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-use-node-status-port
      --metric-resolution=15s  
      --kubelet-insecure-tls ------->

Note: The --kubelet-insecure-tls parameter doesn't verify the CA of serving certificates presented by kubelets. Only use this parameter for testing purposes and not for production clusters. For more information on how to troubleshoot certificate issues, see Why can't I run kubectl commands in Amazon EKS?

Metrics-server pod failed to reach running status and stuck in 0/1 Running

1. If the metric server pod failed to get a running status, then check the connectivity.
   Example:

   NAME                                           READY   STATUS    RESTARTS   AGE
   metrics-server-dbf765b9b-mhqm7                 0/1     Running   0          11m
   

2. Check whether your Metrics Server can scrape the node. If it can't, then you might receive an error message similar to the following:

   "Failed to scrape node, timeout to access kubelet" err="Get \"https://192.168.65.4:10250/metrics/resource\": context deadline exceeded" node="ip-xxx-xxx-xx-xxx.us-west-2.compute.internal" context deadline exceeded" `

   This issue is related to a metrics-server scrape kubelet /metrics/resource or /stats/summary endpoint timeout.

3. Check whether the worker node security group has the required traffic requirements and the port allows 10250 in the cluster security group. If you have multiple virtual private cloud (VPC) CIDR groups attached to the cluster, then make sure to allow all CIDR ranges.

4. If the issue persists, then rollout restart the deployment. Run the following command:

   kubectl rollout restart deployment metrics-server -n kube-system

Related information

Known issues on the GitHub website

Metrics Server on the GitHub website