How do I troubleshoot issues with Amazon EKS managed node groups and launch templates?

Resolution

If you use managed node group launch templates with a specified Amazon Machine Image (AMI) to launch your worker nodes, then you must configure user data. If your cluster is private and uses virtual private cloud (VPC) endpoints to connect, then specify the following attributes in the user data:

• certificate-authority
• api-server-endpoint
• DNS cluster IP address

You must also supply bootstrap commands for worker nodes to join your cluster. If you use Amazon managed node groups with custom launch templates, then specify the correct user data based on your AMI in the launch template.

Note: If you use managed node groups to launch your worker nodes, then you don't need to configure user data for Amazon EKS optimized Amazon Linux AMIs .

Configure the user data for your worker nodes

Configure user data based on your AMI.

AL2

If you use the Amazon Linux 2 AMI, then specify the following information in the user data section of your launch template:

 MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
set -ex
/etc/eks/bootstrap.sh my-cluster \
  --b64-cluster-ca certificate-authority \
  --apiserver-endpoint api-server-endpoint \
  --dns-cluster-ip service-cidr.10 \
  --kubelet-extra-args '--max-pods=my-max-pods-value' \
  --use-max-pods false

--==MYBOUNDARY==-

AL2023

You can use self-managed nodes or managed node groups with a custom launch template. When you create a new node group, you must provide apiServerEndpoint, certificateAuthority, and service CIDR cluster metadata. Amazon Linux uses the additional values to bootstrap the worker node and join the cluster.

Example configuration:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    apiServerEndpoint: https://CCEABD8D413231403775F7AA9E56D037.gr7.us-west-2.eks.amazonaws.com
    certificateAuthority: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJYUxrK2N0M0Zmdll3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBNE1qTXlNREk0TlRaYUZ3MHpOREE0TWpFeU1ETXpOVFphTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURTRWF1c25hS1A5azJYOEd1QkEzQ1czQkZjQ3owSk1EUjRrVDhxUERhOHA4OFY5bkNDcUkvMnlaa2sKSHdIc0hpRDR2OWtoc2hCN1h5OFZaeW56NG8vYkhEZUc5MTZCZ0xzN05Ld1E2Sjd4b0dKc3JyVGtwdUJ3dWpTMgpkbGtLSG0yNTRUelNiQ2VWUkFGbVJGVmljUUh5aEpTSElBN0tUT1dHeUhvUEZJd3l4NS8xZkpmdEIxM0Y3MlBBCm9DaFZBNHdEOVFEZ1NxR0xOMTVMNEN2REFVdFVXaXBtRTZQdWRBNTBxMzNCSzRpRzc4Q0lFbDNXSHl6L3dzOEcKek5hQ2N5ZnlINjRFSHpsVkRkZmFHMWh0ODhKZ01pKy82Vm1vbmthdlJhQzVHZlhsdk5MKzFtVnJycWhTT3hBRQpJS1N2aW9rZndjODA5bzJWaUtiSmhkUE5yQTRKQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJSR3JPdzVaSmJ4ZmI2dTF1ek4wNFJtSkFCU1ZUQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQTFkTnE2aHhXSgp6OU4zM09XamU2cG5LU3pmbGdCaG5PRXRhSUlKa21kUWpUN0FDdlMwMDJyRjFnQUFneTVRR2pjM29mN3hzRzJoCkczemhuUGFTTXFLampsOWRnZGJsZTVEZmIwTjZGVlZXeGE3SFBxZ0NCeXZ1ZXdXOTYwcmxLZ1luMDJBVm16c3kKVW8rRXgwQVA4Q3lVUzQ5Z0RaSFN3MVZGbE1mbFkwb2thSERabG9lR256WDk1REc0Zzk4ZHR1Vkd1Sm1lZ1lpRApKUmN3T2xubUlDUHNsUXcrbGh2S0NFT0RvM0hBWWdwTGxkSFBvYTJCZlBwbk9IMWtMNlFFMURTZU8wQUhmNjBYCk1mTzVTNUwwQVFidk1aM005c2JJdDlhby9YbWk3NnJHMThtMzMrdy9LWkV5cVg1NHZ6UTlZUHBjSGV6bk5oY2oKUy9XajhNZk9KMk9SCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    cidr: 10.100.0.0/16
    name: my-first-eks-cluster
  kubelet:
    config:
      maxPods: 29
      clusterDNS:
      - 10.100.0.10
    flags:
    - "--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=my-first-eks-cluster,alpha.eksctl.io/nodegroup-name=ng-1,eks.amazonaws.com/nodegroup-image=ami-032a2a7895e882f27,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=ng-1,eks.amazonaws.com/sourceLaunchTemplateId=lt-0d8806dd42301728f"

--//--

The nodeadm node initialization process uses a YAML configuration schema. For more information, see nodeadm and Merging multiple configuration objects on the Amazon EKS AMI website.

If you use multiple node configurations in your managed node group launch template, then see Customize managed nodes with launch templates.

Troubleshoot errors in your Amazon EKS clusters

Client.InternalError: Client error on launch

The Client error on launch error occurs when you use encrypted Amazon Elastic Block Store (Amazon EBS) volumes in the custom launch template with incorrect permissions. To check the encryption status and AWS Identity and Access Management (IAM) permissions or policies, complete the following steps:

1. Activate the AWS Key Management Service (AWS KMS) key that AWS KMS uses to encrypt the volume.
2. Make sure that the AWS KMS key has the required key policies.
3. Modify the key policies to allow more IAM roles to work with encrypted Amazon EBS volumes.

SourceEc2LaunchTemplateNotFound or The Amazon EC2 Launch Template lt-abcdefghijklmn version x was not found

The SourceEc2LaunchTemplateNotFound error occurs when you manually change the custom launch template version through the node's Amazon EC2 Auto Scaling group. You must use the Amazon EKS console to update the launch template version.

To update your Amazon Elastic Compute Cloud (Amazon EC2) launch template from the managed node group, complete the following steps:

1. Open the Amazon EKS console.
2. In the navigation pane, choose Clusters.
3. Under Cluster name, choose the cluster with the node group that you want to update.
4. Choose the Compute tab.
5. For Node groups, select your node, and then choose Change version.
6. Select the updated node group version. Also, make sure that Update strategy is Rolling update.
7. Choose Update.

This error also occurs when you use a launch template to create a node group and you delete a reference template. Amazon EKS replicates the template that you use to create the node group. If you delete the original template, then you must recreate the node group.

To recreate the node group, complete the following steps:

1. Launch a new node group.

2. Run the following command to verify that all nodes are healthy, in the Ready state, and joined to the cluster:

   kubectl get nodes

3. Drain your worker nodes.

4. Delete the original node group.

The Node group enters a DEGRADED status after it creates a new launch template

If a node group enters a DEGRADED status, then you receive a message that's similar to the following example:

"The Amazon EC2 Launch Template : lt-abcdefghijklmn has a new version associated with your Autoscaling group, which is not managed by Amazon EKS. Expected Launch Template version: x".

This error occurs when the Amazon EC2 launch template version for your managed node group doesn't match the version that Amazon EKS creates.

You can't directly update existing node groups that don't use a custom launch template. To resolve this issue, create a launch template and version with your custom settings. Then, use the launch template to create the node group. If you launched the new node group from your custom template, then create a new version of the template.

Instance failed to join the Amazon EKS cluster

To resolve this issue, see How do I get my worker nodes to join my Amazon EKS cluster?