How do I troubleshoot custom launch template issues with managed node groups in Amazon EKS?

Short description

You might get one of the following errors when using a custom launch template with managed node groups in your Amazon EKS cluster:

• "Client.InternalError: Client error on launch"
• "SourceEc2LaunchTemplateNotFound or The Amazon EC2 Launch Template lt-xxxxxxxxxxxxxxxxx version x was not found"
• "Node group entered a DEGRADED status"

Resolution

Client.InternalError: Client error on launch

This error occurs when using encrypted Amazon Elastic Block Store (Amazon EBS) volumes in the custom launch template with incorrect permissions. To check the encryption status and AWS Identity and Access Management (IAM) permissions or policies, verify the following:

• The AWS Key Management Service (AWS KMS) key used to encrypt the volume is activated.
• The AWS KMS key has the correct key policies. To allow more IAM roles to work with encrypted Amazon EBS volumes, you can modify the key policies. For more information, see Allows key users to use the KMS key.

SourceEc2LaunchTemplateNotFound or The Amazon EC2 Launch Template lt-xxxxxxxxxxxxxxxxx version x was not found

This error occurs when you manually change the custom launch template version through the Auto Scaling group of the node instead of Amazon EKS. To resolve this issue, you must update the launch template version using Amazon EKS.

To update your EC2 launch template from the managed node group, complete the following steps:

1. Open the Amazon EKS console.
2. In the navigation pane, choose Clusters.
3. In Cluster name, choose the cluster that contains the node group to update.
4. Choose the Compute tab.
5. For Node groups, choose your node and then choose Change version.
6. Select the version to apply to your node group. Make sure that the update strategy is set to Rolling Update.
7. Choose Update.

For more information, see Updating a managed node group.

Node group entered a DEGRADED status after creating a new launch template

Node group can enter a DEGRADED status with an error similar to the following:

"The Amazon EC2 Launch Template : lt-xxxxxxxxxxxxxxxxx has a new version associated with your Autoscaling group, which is not managed by Amazon EKS. Expected Launch Template version : x".

This issue happens when the Amazon EC2 launch template version for your managed node group doesn't match the version that Amazon EKS created. Existing node groups that don't use a custom launch template can't be updated directly. To resolve this, create a launch template and version with your preferred settings, and then create the node group using that launch template. If the new node group is launched from your custom template, then you can create new versions of the template. You can use this template without putting the node group in a DEGRADED status.

Related information

Launch template support

Launch template configuration basics