How does EKS decides which node to be removed during node group resizing

Question

Hi there,

For an EKS cluster without auto-scaling enabled, if we scaled down a node group by updating the node group config (decreasing the desired node size). How does EKS decide which node to be removed?

If the removing algorithm is just random, is there any way to enforce a specific node to be removed? If we drain and cordon a node to make it unscheduled, will EKS favour this and delete it?

Thanks!

Answer

Hello, greetings for the day.

I see that you are interested in knowing which node (instance) would be taken out (removed) first if you are using nodgroups in EKS cluster.

As you might already know that the nodegroups[1] utilize autoscaling groups at the back end of a nodegroup and the selection of the instance that needs to be terminated depends on the autoscaling group termination policies.

Suppose you have a nodegroup that has a few instances running and if you decrease the desired size of the nodegroup by 1 then the oldest running instance would be selected to terminate first.  You can verify this yourself by going to the autoscaling group created by EKS for your nodegroups. On the autosclaing group console, under advanced configurations, there is section called as termination policies and here you will see the following three entries:  
-AllocationStrategy: This comes into picture when there are both spot and on demand instances.  -OldestLaunchTemplate: Which means the instances launched by the older launch template will be terminated first.  
-OldestInstance: Which means the oldest instance will be terminated first.

To answer your question: " If we drain and cordon a node to make it unscheduled, will EKS favour this and delete it?"  
Answer: No, the instance that gets terminated would depend on the termination policies we have discussed above.

Please refer this documentation[2] for more information on what I have shared above.

The answer(s) that I have provided are considering the autoscaling group itself and not including the part where the cluster autoscaler[3][4] plays a role.

Reference:  
[1] https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html  
[2]  https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-termination-policies.html#custom-termination-policy

[3] https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md  
[4] https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler

How does EKS decides which node to be removed during node group resizing

Relevant content