Why can't I use Metrics Server in Amazon EKS to collect metrics from containers, pods, or nodes?

Resolution

Amazon EKS doesn't automatically install Metrics Server. If you recently created your cluster and can't use Metrics Server to collect metrics, then confirm that you deployed the Metrics Server application to your cluster.

If you still can't collect metrics with Metrics Server, then complete the steps in the following sections.

Note: Metrics Server isn't a best practice for long-term monitoring of application and cluster performance. For long-term monitoring, see Resource management for pods and containers on the Kubernetes website. The Kubernetes community maintains Metrics Server and reports issues on its GitHub page. For more information see, Issues on the Metrics Server GitHub page.

Check if you can retrieve metrics from your cluster's nodes and pods

To check whether you can retrieve metrics from your cluster's nodes and pods, run the following commands:

kubectl top nodes
kubectl top pods

Note: If you don't receive an error from either command, then refer to the Check if the APIService is available and can handle requests section.

If you receive an error from the preceding command, then complete the steps in one of the following sections based on the error that you receive.

"Error from server (Forbidden)" error message

If you have an issue with role based access control (RBAC) authorization, then you receive an "Error from server (Forbidden)" error message.

To resolve this error, take the following actions:

• Confirm that the ServiceAccount correctly attaches to the deployment.
• Verify that the ClusterRole/Role and ClusterRoleBinding/RoleBindings use the correct RBAC permissions for Metrics Server. For more information, see Using RBAC authorization on the Kubernetes website.

If you access your cluster through a role defined in the aws-auth ConfigMap, then confirm that you set the username field and the mapping.

Complete the following steps:

1. To describe the aws-auth ConfigMap, run the following command:

   kubectl describe -n kube-system configmap aws-auth

2. Review the output from the preceding command, to confirm that the username field is set for the role that accesses the cluster.
   Example output:

   Name:         aws-auth
   Namespace:    kube-system
   Labels:       
   Annotations:  <none>
   
   Data
   ===
   mapRoles:
   ---- 
   ...
   - groups:
     - system:masters
     rolearn: arn:aws:iam::123456789123:role/kubernetes-devops
     username: devops:`SessionName`

   Make sure that the username is specified in the output.

"Error from server (ServiceUnavailable)" error message

If Metrics Server is not properly configured, then you receive the "Error from server (ServiceUnavailable)" error message.

To check for an issue with the configuration of the Metrics Server service application in your cluster, run the following command:

kubectl describe apiservices v1beta1.metrics.k8s.io

Example output:

Name:         v1beta1.metrics.k8s.io
Namespace:
Labels:       app=metrics-server
...
Status:
  Conditions:
    Last Transition Time:  2020-01-09T13:57:23Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

If the Metrics Server service is available and passes checks, then Status is set to True.

Note: If you set Status to True and the issue persists, then refer to the Check if the APIService is available and can handle requests section.

If Status is set to False, then look for the associated Reason code and human readable Message for Conditions in the output.

Failing APIService example:

...
Status:
  Conditions:
    Last Transition Time:  2020-01-09T14:40:28Z
    Message:               no response from https://10.0.35.231:443: Get https://10.0.35.231:443: dial tcp 10.0.35.231:443: connect: connection refused
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available

Note: If the Reason isn't FailedDiscoveryCheck, then refer to the Other APIServer condition failure reasons section. If the APIServer Conditions Message contains Client.Timeout exceeded while awaiting headers, then refer to the Resolve the "Client.Timeout exceeded while awaiting headers" error section. If the APIServer Conditions Message contains Connection refused, then refer to the Resolve the "Connection refused" error section.

Resolve the "Client.Timeout exceeded while awaiting headers" error

If you didn't correctly configure a security group or network access control list (network ACL), then you receive the "Client.Timeout exceeded will awaiting headers" error message on the APIService. Then, you can't access the metrics-server pods.

To resolve this error, confirm that your security groups comply with minimum traffic requirements for Amazon EKS.

Resolve the "Connection refused" error

If a container listens on the wrong port, then you receive the "Connection refused" error message.

To resolve this error, run the following command to confirm that the values for ports, image, and command are correct in the metrics-server deployment:

kubectl describe deployment metrics-server -n kube-system

Example output:

Name:                   metrics-server
Namespace:              kube-system
CreationTimestamp:      Wed, 08 Jan 2020 11:48:45 +0200
Labels:                 app=metrics-server
...
  Containers:
   metrics-server:
    Image:      gcr.io/google_containers/metrics-server-amd64:v0.3.6
    Port:       443/TCP
    Command:
    - /metrics-server
    - --logtostderr
    - --secure-port=443
...

Note: The Command and Image values vary depending on how you deploy the Metrics Server and where you store the images. If the Command contains the --secure-port parameter, then the port exposed by the pod must match this parameter. If the Command doesn't include the --secure-port parameter, then the port defaults to 443.

Other APIServer condition failure reasons

If you receive any of the following codes for the APIService, then take action based on the associated error message: ServiceNotFound, ServiceAccessError, ServicePortError, EndpointsNotFound, EndpointsAccessError, or MissingEndpoints.

To resolve the errors, complete the following steps:

1. To get information on the service with the error, run the following command:

   kubectl get service -n kube-system

   In the output, confirm that the Kubernetes service has the same name and namespace as defined in APIService.Spec.Service. Then, confirm that the port is set to 443/TCP.
   Example output:

   NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   metrics-server                   ClusterIP   172.20.172.133   <none>        443/TCP    65m

2. Run the following command to list all the endpoints:

   kubectl get endpoints metrics-server -n kube-system

   In the output, confirm that you have at least one endpoint for the metrics-server service.
   Example output:

   NAME             ENDPOINTS         AGE
   metrics-server   10.0.35.231:443   76m

3. Run the following command to confirm that the deployment is present and that the labels match those of the metrics-server service:

   kubectl describe deploy metrics-server -n kube-system

   In the output, confirm that the deployment has at least one replica.
   Example output:

   Name:                   metrics-server
   Namespace:              kube-system
   CreationTimestamp:      Wed, 08 Jan 2020 11:48:45 +0200
   Labels:                 app=metrics-server
                           release=metrics-server
   ...
   Selector:               app=metrics-server,release=metrics-server
   Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
   ...
   Pod Template:
     Labels:           app=metrics-server
                       release=metrics-server
     Service Account:  metrics-server
     Containers:
      metrics-server:
       Image:      gcr.io/google_containers/metrics-server-amd64:v0.3.6
   ...
   

If you still can't collect metrics with Metrics Server, then refer to the Check if the APIService is available and can handle requests section.

Check if the APIService is available and can handle requests

Run the following command to extract logs from your Metrics Server pods:

kubectl logs -n namespace -l app=metrics-server

Note: Replace namespace with your namespace.
Example error logs:

E0610 23:13:28.247604       1 reststorage.go:98] unable to fetch pod metrics for pod default/php-apache-b5f58cc5f-nv8sz: no metrics known for pod "default/php-apache-b5f58cc5f-nv8sz"
E0610 23:13:43.260069       1 reststorage.go:98] unable to fetch pod metrics for pod default/php-apache-b5f58cc5f-nv8sz: no metrics known for pod "default/php-apache-b5f58cc5f-nv8sz"
E0610 23:16:13.346070       1 reststorage.go:98] unable to fetch pod metrics for pod default/php-apache-b5f58cc5f-cj67b: no metrics known for pod "default/php-apache-b5f58cc5f-cj67b"
E0610 23:16:13.346087       1 reststorage.go:98] unable to fetch pod metrics for pod default/php-apache-b5f58cc5f-sqc6l: no metrics known for pod "default/php-apache-b5f58cc5f-sqc6l"
E0610 23:16:13.346091       1 reststorage.go:98] unable to fetch pod metrics for pod default/php-apache-b5f58cc5f-4cpwk: no metrics known for pod "default/php-apache-b5f58cc5f-4cpwk"

Note: Metrics Server error logs show if you have a configuration issue on the Metrics Server Deployment Command or a bug with the Metrics Server container. If the error message isn't obvious or you suspect that it's a bug, then complete the steps in the Check GitHub for common issues section.

Check GitHub for common issues

If you still can't collect metrics from containers, pods, or nodes, then check GitHub for common issues with Metrics Server. For more information, see Issues on the Metrics Server GitHub page.

To check your HorizontalPodAutoscaler (HPA) and application resource requests for unknown metrics, complete the following steps:

1. Run the following command to check the HPA configuration:

   kubectl get hpa -n namespace 2048-deployment

   Note: Replace namespace and 2048-deployment with the HPA configuration values for your application.
   You might see <unknown> under the Targets column of the output.
   Example output:

   NAME              REFERENCE                    TARGETS         MINPODS   MAXPODS   REPLICAS   AGE  
   2048-deployment   Deployment/2048-deployment   <unknown>/80%   1         2         2          10s

2. Wait several minutes, and then repeat the command from step 1.
   If you still receive the "<unknown>" error, then run the following command:

   kubectl describe hpa -n namespace 2048-deployment

   Note: Replace namespace with your namespace.
   Then, check the Events section of the output for more information.
   Example output:

   Name:                                                  2048-deployment
   Namespace:                                             2048-game
   ...
   Metrics:                                               ( current / target )
     resource cpu on pods  (as a percentage of request):  <unknown> / 80%
   Min replicas:                                          1
   Max replicas:                                          2
   Deployment pods:                                       2 current / 2 desired
   Conditions:
     Type           Status  Reason                   Message
     ----           ------  ------                   -------
     AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
     ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for cpu
   Events:
     Type     Reason                   Age                     From                       Message
     ----     ------                   ----                    ----                       -------
     Warning  FailedGetResourceMetric  3m29s (x4333 over 19h)  horizontal-pod-autoscaler  missing request for cpu

   Note: If the Message column shows missing request for [x], then your Deployments or ReplicaSet might not declare resource requests in its specification. Confirm that all the containers in the pod have the requests declared. If you leave out a request, then the metric in HPA returns the <unknown> response.
   For more information, see Resource management for pods and containers, Deployments, and ReplicaSet on the Kubernetes website.