Inquiries about NTH works

Hello,

Greetings for the day!!

From your correspondence I can understand that you are using karpenter to provision Spot nodes on your EKS cluster and you are planning to use Node Termination Handler (NTH) and you would like to confirm if you need to set the 'enableSpotInterruptionDraining' and 'enableRebalanceDraining' to false in order to avoid recursive creation and removal of Spot nodes. You would also like to know the scenarios under which recursive creation and removal of node happens. Please correct me if I misunderstood anything.

Yes, you are correct. If you are using Spot instances on your EKS cluster and also employ Karpenter for provisioning nodes then you are recommended to set the aforementioned values to false.

Scenarios in which recursive creation and removal happens:

• Let us say you are using an EKS cluster with 5 EC2 worker node instances and out of them 3 are On Demand instances and 2 are Spot instances.
• Now you are using Karpenter to provision Spot nodes in your cluster and you also have a Node Termination Handler (NTH) configured.
• Now your Spot worker nodes instances receive a Spot interruption notification or a Spot rebalance recommendation and in both of these cases the NTH may remove the Spot node(s) and the pods that were running on these nodes may trigger karpenter to launch more nodes so that those pods can run and the Karpenter may again end up launching more Spot worker node instances which will again be removed by the NTH due the same reason as before and this cycle may continue.

In short, the recursive creation/removal of nodes will happen if:

• If you are using karpenter to provision nodes and this is applicable if Karpenter is being used to provision Spot nodes.
• You are using NTH with values for 'enableSpotInterruptionDraining' and 'enableRebalanceDraining' NOT set to false.

I hope the above sheds light on the scenario when recursive creation and removal of nodes happen.

Have a fantastic day ahead!!