Auto scaling group in EKS with encrypted EBS volumes

0

Hello, We have enforced EBS encryption to true by default in AWS Dashboard. However after changing this config, we are getting the below error on launching Instances on Node group.

Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state

Any idea , How to enforce ASG with EBS encrytion turned on.

2 Answers
0

Hi. Check if KMS key is enabled?

profile picture
answered a year ago
  • Yes, KMS keys are enabled. I believe for ASG , we need to specify Launch template with kms keys. but not sure how to do it. Do you have an idea on it.

0

Hello,

If you want to encrypt Amazon EBS volumes for your nodes, you can deploy the nodes using a launch template. To deploy managed nodes with encrypted Amazon EBS volumes without using a launch template, encrypt all new Amazon EBS volumes created in your account. I see you have already enabled encryption by default. If not, you can use a custom launch templates and provide KMS key details for encrypting volumes. The same can be found in documentation [1].

If you plan to create node groups with custom launch templates you can refer to the documentation [2]. When you create your launch template, in the "Storage (volumes)" section you can add a volume with desired configurations - enable the encryption and choose the required KMS key. You can then use the launch template to create the nodegroup (update if its already using a custom launch template).

When you use a customer-managed key for launching instances with encrypted volumes using ASG, you need to ensure that the key policy section of the key gives the service-linked role - AWSServiceRoleForAutoScaling (service-linked role for Amazon EC2 Auto Scaling) permissions to use the customer managed key. You can refer "Example 1: Key policy sections that allow access to the customer managed key" in the documentation [3] for reference. Additional details about this can be referred from the same document.

If everything is configured in the launch template and key policy has correct permissions, you can check the below things: (i) Ensure that the key exists and is enabled. (ii) Check if there are any errors/failed API calls related to KMS actions in the CloudTrail. If you find any, you can add required permissions based on the error received.

You can also refer the article [4] which talks about the "must-know best practices for Amazon EBS encryption".

If the error still persists after making the changes by referring these documents and the above suggestions, we require details that are non-public information, that is, we might need to check the cluster, ASG, launch template and other resources. Please open a support case with AWS using the link - https://console.aws.amazon.com/support/home#/case/create

[1] https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html#managed-node-group-concepts

[2] https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html

[3] https://docs.aws.amazon.com/autoscaling/ec2/userguide/key-policy-requirements-EBS-encryption.html

[4] https://aws.amazon.com/blogs/compute/must-know-best-practices-for-amazon-ebs-encryption/

AWS
answered a year ago
  • Thank you for the answer, Does auto scaling group use the same launch template as managed node groups. I see ASG alone is failing with the KMS error, Nodes initially launched are working fine.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions