How to remove a specific node from EKS node group (none managed eksctl)

Question

I have a small EKS cluster created by eksctl. It consists of 2 node groups (none managed).

The first group runs the management app and currently consists of a single node, the second group runs the "worker" pods and has 3 nodes.

I have received an event notification that the EC2 machine that is the single node in the management nodegroup is scheduled to be shutdown due to maintenance. My plan is to scale the group to 2 nodes and then use `kubectl drain` to evict the running application so it will move to the new node. All good so far.

My question is how to then remove the original node before EC2 shuts it down (and presumably starts a replacement)? Is there a way to specify a specific node when scaling the group back down to a single node with eksctl? Or will it be smart enough to pick the node that is already drained/cordoned?

Accepted Answer

I understand one of the EKS node is scheduled for the maintenance. Currently you have second group that runs the "worker" pods and has 3 nodes. You want to scale in the group to 2 nodes and want the node scheduled for the maintenance should are removed as part of scale in. You have workload already transferred to other 2 node at moment.

Below is list of steps I have tested in my test environment to achieve this.

You might already be aware you have autoscaling group for the group and it takes care of launching/termination of node and keep the node count as per you autoscaling desired capacity.

- Drain on EKS:

$ kubectl get nodes

$ kubectl cordon

$ kubectl drain  --ignore-daemonsets

To terminate the instance and also decrements the size of the Auto Scaling group.

$ aws autoscaling terminate-instance-in-auto-scaling-group --instance-id  --should-decrement-desired-capacity --region

Refer: https://docs.aws.amazon.com/cli/latest/reference/autoscaling/terminate-instance-in-auto-scaling-group.html

How to remove a specific node from EKS node group (none managed eksctl)

관련 콘텐츠