Hello,
I've had problem to scale my EKS Cluster. When I try add new node, I get the errors below and de nodegroup shows a message "Degrated".
Error on Nodegroup, on Health Issues: AsgInstanceLaunchFailures (Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch)
Error on Auto Scaling Group: (Launching a new EC2 instance: i-[instancename]. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch)
**Error in Cloudtrail: **
"invokedBy": "eks-nodegroup.amazonaws.com"
},
"eventTime": "2023-08-26T13:48:55Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "RunInstances",
"awsRegion": "sa-east-1",
"sourceIPAddress": "eks-nodegroup.amazonaws.com",
"userAgent": "eks-nodegroup.amazonaws.com",
"errorCode": "Client.DryRunOperation",
"errorMessage": "Request would have succeeded, but DryRun flag is set.",
"requestParameters": {
"instancesSet": {
"items": [
{
"minCount": 1,
"maxCount": 1
}
]
},
"instanceType": "t3a.xlarge",
"blockDeviceMapping": {},
"availabilityZone": "sa-east-1a",
"monitoring": {
"enabled": false
},
"subnetId": "subnet-099dechanged02c7a1f",
"disableApiTermination": false,
"disableApiStop": false,
"clientToken": "fa5f1139-0946-changed-959a-4a3935f58d3c",
"launchTemplate": {
"launchTemplateId": "lt-09a9changed30c25e",
"version": "3"
}
So, I already try:
-
Scale the actual nodegroup from 3 for 4, with the actual Lauch Template and several versions, with some changes, like image, disks etc. I got the same errors.
-
Create a new nodegroup with same Lauch Template, but I receive "Creation Failed", with the same erros.
-
Create a new nodegroup with NEW Lauch Template, but I receive "Creation Failed", with the same erros.
-
I checked the permissions again and they are ok. I confirm the permissions to KMS, again OK. The key in KMS allows AWS Managed policy for EKS, like I saw in many documentations. Also, I set user and role that de nodegroup uses to access the Key in KMS, but always I have the same error.
Particularly, I thing that "dry run" option isn't works with RunInstance on EKS, to add a node to a nodegroup, but I don't know how to disable it.
Could someone help me in this issue?
Hello Shadad!
Thank you for your time do answer my question!
I already insert the correct policy in KMS Key, but, I saw now in CloudTrail that I was setting the policy in a wrong KMS. CloudTrail give the correct key that I've had problem, so I put the policy in the key and I can now Scaling my node cluster.
Thank you for you help me! The tips about Cloudtrail was the key to identify the problem. In my case, there are key to EKS and key to EBS, I need to set the correct policy in this EBS Key.
Thank You so Much!
Glad that resolved your issue. And thank you for the feedback + confirmation :D