EKS Cluster - I cannot add a new node to a nodegroup. (Cloudtrail Error: Client.DryRunOperation). (EKS Nodegoup Error: AsgInstanceLaunchFailures). (ASG error: Client.InternalError: Client error on lau

Question

Hello,
I've had problem to scale my EKS Cluster. When I try add new node, I get the errors below and de **nodegroup **shows a message** "Degrated".**
**Error on Nodegroup,** on Health Issues: AsgInstanceLaunchFailures (Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch)
**Error on Auto Scaling Group:** (Launching a new EC2 instance: i-[instancename]. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch)
**Error in Cloudtrail: **
    "invokedBy": "eks-nodegroup.amazonaws.com"
    },
    "eventTime": "2023-08-26T13:48:55Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "RunInstances",
    "awsRegion": "sa-east-1",
    "sourceIPAddress": "eks-nodegroup.amazonaws.com",
    "userAgent": "eks-nodegroup.amazonaws.com",
    "errorCode": "Client.DryRunOperation",
    "errorMessage": "Request would have succeeded, but DryRun flag is set.",
    "requestParameters": {
        "instancesSet": {
            "items": [
                {
                    "minCount": 1,
                    "maxCount": 1
                }
            ]
        },
        "instanceType": "t3a.xlarge",
        "blockDeviceMapping": {},
        "availabilityZone": "sa-east-1a",
        "monitoring": {
            "enabled": false
        },
        "subnetId": "subnet-099dechanged02c7a1f",
        "disableApiTermination": false,
        "disableApiStop": false,
        "clientToken": "fa5f1139-0946-changed-959a-4a3935f58d3c",
        "launchTemplate": {
            "launchTemplateId": "lt-09a9changed30c25e",
            "version": "3"
        }

So, I already try:

- Scale the actual nodegroup from 3 for 4, with the actual Lauch Template and several versions, with some changes, like image, disks etc. I got the same errors.
- Create a new nodegroup with same Lauch Template, but I receive "Creation Failed", with the same erros.
- Create a new nodegroup with NEW Lauch Template, but I receive "Creation Failed", with the same erros.

- I checked the permissions again and they are ok. I confirm the permissions to KMS, again OK. The key in KMS allows AWS Managed policy for EKS, like I saw in many documentations. Also, I set user and role that de nodegroup uses to access the Key in KMS, but always I have the same error.

Particularly, I thing that "dry run" option isn't works with RunInstance on EKS, to add a node to a nodegroup, but I don't know how to disable it.

Could someone help me in this issue?

Accepted Answer

AutoScaling is the one launching the instances, not EKS.  So you need to make sure that AutoScaling has access to the key
https://docs.aws.amazon.com/autoscaling/ec2/userguide/key-policy-requirements-EBS-encryption.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-launch.html#troubleshooting-launch-internal-solution

If the key is cross-account, you'll also need to make sure to make the create-grant call listed in the doc *from the ASG account*

To confirm the correct key is in use, check CloudTrail for EventSource=kms.amazonaws.com around the time of the failures and see if there's any calls erroring out.  These will likely have info on the specific key being used.

---

For the DryRun call, check the specific error message, if its something like "Would have succeeded, but DryRun flag was set", then you can ignore it.  That's a "good" error and is AutoScaling validating that you have permissions to launch the instances which the group is configured to use.

EKS Cluster - I cannot add a new node to a nodegroup. (Cloudtrail Error: Client.DryRunOperation). (EKS Nodegoup Error: AsgInstanceLaunchFailures). (ASG error: Client.InternalError: Client error on lau

Relevant content