Karpenter Consolidation (PodDisruptionBudget) Application downtime

0

Hello, I have an EKS cluster with one node specifically for Karpenter and Karpenter creates worker nodes based on the requirements of the deployment of my application. The issue arises when the pods scale out. When the pods scale out due to a rise in utilization, Karpenter creates nodes based on the nodepool configuration that I have set. After it creates the nodes and new pods are scheduled to those nodes, Karpenter finds a better EC2 instance type and creates a new node and deletes all the pods/nodes that exist causing downtime in my application.

I have tried setting the PodDisruptionBudget so that there is always 1 pod available. While this does work when scaling out, the problem is that when it is time to scale in and change the instance type, Karpenter does not change the instance type with the reason that the PodDisruptionBudget is preventing this.

An example of the above scenario is this:

  1. t3.medium with 1 Pod
  2. Increase in load occurs
  3. number of Pods increases to 5
  4. Karpenter creates 3 t3.medium nodes
  5. While Pods are getting ready and initializing (typically takes 4 minutes), Karpenter decides to change the 4 t3.medium nodes to 1 t3.2xlarge nodes
  6. Because of the above, all pods are terminated (unless the PodDisruptionBudget is set) and causes downtime.
  7. In the case that the PDB is set, there is no down time but when the load decreases and the number of pod is reduced to 1 or 2, we end up with 1 t3.2xlarge for 1 or 2 pods and Karpenter does not change the instance type because the PDB is preventing the consolidation.

Question: Is there a way to make sure that Karpenter freely changes instances types and creates nodes during load increase and decrease AND also make sure that at least 1 pod is available/running to ensure that there is no downtime?

Thank you

John
asked 5 months ago267 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions