Why are On-Demand nodes being replaced in my EKS cluster with Karpenter, causing pod restarts?

Question

I have an EKS cluster running in auto mode. In the node pool, I’ve configured both Spot and On-Demand instances. Only for the production environment, I’ve set a node selector: karpenter.sh/capacity-type=on-demand. However, I’m noticing that even On-Demand instances are being replaced — new nodes are added while some existing ones are removed. Due to this, my production pods are being moved to new nodes, which may cause downtime. These are still On-Demand instances, but I'm unsure why this replacement is happening. What’s the reason behind removing some and adding new ones?

Marouen Ouenzerfi · Answer

In addition to EKS Auto Mode automatic node patching, disruption can happen because of karpenter consolidation.

You can influence that by configuring Pod Disruption Budgets or the disruption budget in the NodePool configuration. Additionally, if you want to disable deprovisioning a node for a specific pod , you can use the karpenter.sh/do-not-disrupt annotation.

https://docs.aws.amazon.com/eks/latest/best-practices/karpenter.html

Ryan Stebich · Answer

Nodes launched by EKS Auto Mode have a maximum lifetime of 21 days (Default 14 days), after which they are automatically replaced with new nodes. This approach enhances your security posture by regularly cycling nodes, aligning with best practices already adopted by many customers.

If you are seeing your nodes replaced on a regular cycle i.e. every 14 days this is likely the cause.

https://docs.aws.amazon.com/eks/latest/userguide/automode.html

If you are seeing your nodes replaced on a regular cycle i.e. every 14 days this is likely the cause.

https://docs.aws.amazon.com/eks/latest/userguide/automode.html

Why are On-Demand nodes being replaced in my EKS cluster with Karpenter, causing pod restarts?

Add your answer

Relevant content