EKS Cluster - I cannot add a new node to a nodegroup. (Cloudtrail Error: Client.DryRunOperation). (EKS Nodegoup Error: AsgInstanceLaunchFailures). (ASG error: Client.InternalError: Client error on lau

Question

Hello,
I've had problem to scale my EKS Cluster. When I try add new node, I get the errors below and de **nodegroup **shows a message** "Degrated".**
**Error on Nodegroup,** on Health Issues: AsgInstanceLaunchFailures (Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch)
**Error on Auto Scaling Group:** (Launching a new EC2 instance: i-[instancename]. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch)
**Error in Cloudtrail: **
    "invokedBy": "eks-nodegroup.amazonaws.com"
    },
    "eventTime": "2023-08-26T13:48:55Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "RunInstances",
    "awsRegion": "sa-east-1",
    "sourceIPAddress": "eks-nodegroup.amazonaws.com",
    "userAgent": "eks-nodegroup.amazonaws.com",
    "errorCode": "Client.DryRunOperation",
    "errorMessage": "Request would have succeeded, but DryRun flag is set.",
    "requestParameters": {
        "instancesSet": {
            "items": [
                {
                    "minCount": 1,
                    "maxCount": 1
                }
            ]
        },
        "instanceType": "t3a.xlarge",
        "blockDeviceMapping": {},
        "availabilityZone": "sa-east-1a",
        "monitoring": {
            "enabled": false
        },
        "subnetId": "subnet-099dechanged02c7a1f",
        "disableApiTermination": false,
        "disableApiStop": false,
        "clientToken": "fa5f1139-0946-changed-959a-4a3935f58d3c",
        "launchTemplate": {
            "launchTemplateId": "lt-09a9changed30c25e",
            "version": "3"
        }

So, I already try:

- Scale the actual nodegroup from 3 for 4, with the actual Lauch Template and several versions, with some changes, like image, disks etc. I got the same errors.
- Create a new nodegroup with same Lauch Template, but I receive "Creation Failed", with the same erros.
- Create a new nodegroup with NEW Lauch Template, but I receive "Creation Failed", with the same erros.

- I checked the permissions again and they are ok. I confirm the permissions to KMS, again OK. The key in KMS allows AWS Managed policy for EKS, like I saw in many documentations. Also, I set user and role that de nodegroup uses to access the Key in KMS, but always I have the same error.

Particularly, I thing that "dry run" option isn't works with RunInstance on EKS, to add a node to a nodegroup, but I don't know how to disable it.

Could someone help me in this issue?

Shahad_C · Accepted Answer

AutoScaling is the one launching the instances, not EKS.  So you need to make sure that AutoScaling has access to the key
https://docs.aws.amazon.com/autoscaling/ec2/userguide/key-policy-requirements-EBS-encryption.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-launch.html#troubleshooting-launch-internal-solution

If the key is cross-account, you'll also need to make sure to make the create-grant call listed in the doc *from the ASG account*

To confirm the correct key is in use, check CloudTrail for EventSource=kms.amazonaws.com around the time of the failures and see if there's any calls erroring out.  These will likely have info on the specific key being used.

---

For the DryRun call, check the specific error message, if its something like "Would have succeeded, but DryRun flag was set", then you can ignore it.  That's a "good" error and is AutoScaling validating that you have permissions to launch the instances which the group is configured to use.

EKS Cluster - I cannot add a new node to a nodegroup. (Cloudtrail Error: Client.DryRunOperation). (EKS Nodegoup Error: AsgInstanceLaunchFailures). (ASG error: Client.InternalError: Client error on lau

Relevant content