EKS with karpenter and balance node in 2 AZ

Question

Hi everyone,

I have an EKS 1.32 cluster running several applications. I'm heavily using Karpenter with EC2 SPOT instances.
My Karpenter nodepools are correctly configured to use both Availability Zones A and B in the Milan region.
My VPC has 2 AZ. I'm not use EKS automode and no fargate.

However, I’ve noticed that almost every time Karpenter launches a new node, it’s placed in Zone B.
Currently, I have 13 EKS nodes: 11 in Zone B and only 2 in Zone A. I believe this distribution is too unbalanced.

I’ve read that setting up Topology Spread Constraints could help improve balance, since Karpenter tends to launch nodes where EC2 costs are lower or availability is higher. If I understand correctly, Topology Spread Constraints help distribute pods in a deployment, but many of my applications consist of a single pod, so this feature is not very effective in my case (open to suggestions).

My question:
Is there a way to achieve a better cluster-level AZ distribution automatically, without manually creating Karpenter nodepools limited to a specific zone (which would be a manual workaround rather than a dynamic solution)?

This is my karpenter nodepool config

```

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: standard
spec:
  template:
    metadata:
      labels:
        node-scope: standard
    spec: 
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64", "amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: ["eu-south-1a", "eu-south-1b"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

```

Thanks in advance!

Bruno Lopes · Answer

Have you ever tried to set up Topology Spread using parameters **maxSkew** and **minDomain**, to achieve this balance?

I'm doing some tests on it and achieving exactly this scenario. Look at this:

```
topologySpreadConstraints:
- maxSkew: 1
   minDomains: 3
   topologyKey: topology.kubernetes.io/zone
   whenUnsatisfiable: DoNotSchedule
   labelSelector:
       matchLabels:
           app: my-app
```

re:Post Agent · Answer

The imbalance you're experiencing with Karpenter placing most nodes in Zone B is likely because Karpenter is optimizing for cost and availability rather than zone balance. By default, Karpenter will provision nodes as quickly and cheaply as possible, which may lead to concentration in a single zone if that zone has better pricing or availability for Spot instances.

To achieve better AZ distribution, you have a few options:

1. **Use Pod Topology Spread Constraints**: Even for single-pod applications, you can use topology spread constraints in your pod specifications. This would encourage the Kubernetes scheduler to spread pods across zones, which would in turn cause Karpenter to provision nodes in different zones. For example:

```yaml
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: "topology.kubernetes.io/zone"
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: your-app-label
```

2. **Create Multiple NodePools with Zone Constraints**: While you mentioned wanting to avoid this approach, creating separate NodePools for each zone with appropriate capacity limits can help ensure a more balanced distribution. This isn't entirely manual as Karpenter will still handle the scaling within each zone.

3. **Adjust Workload Requirements**: Your workloads can specify their availability requirements through taints, tolerations, and PodTopologySpread. By defining these requirements, you can influence how Karpenter provisions nodes across zones.

4. **Consider Using EKS Auto Mode**: Although you mentioned you're not using it, EKS Auto Mode builds upon Karpenter and can help with more balanced zone distribution.

The key insight is that Karpenter is primarily focused on meeting workload requirements efficiently rather than ensuring zone balance. By explicitly defining zone distribution requirements in your workloads, you can guide Karpenter to provision nodes more evenly across zones.  
**Sources**  
[Kubernetes Data Plane - Amazon EKS](https://docs.aws.amazon.com/eks/latest/best-practices/scale-data-plane.html)  
[Scale cluster compute with Karpenter and Cluster Autoscaler - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html)  
[VPC and Subnet Considerations - Amazon EKS](https://docs.aws.amazon.com/eks/latest/best-practices/subnets.html)  
[Community | Run Kubernetes Clusters for Less with Amazon EC2 Spot and Karpenter](https://community.aws/content/2dhlDEUfwElQ9mhtOP6D8YJbULA/run-kubernetes-clusters-for-less-with-amazon-ec2-spot-and-karpenter)

EKS with karpenter and balance node in 2 AZ

Add your answer

Relevant content