Skip to content

AZ imbalance with equal NodePool weights for Spot instances (ARM64)

0

Summary

I’m trying to have Karpenter evenly launch Spot nodes across two Availability Zones (eu-south-1a and eu-south-1b).
To achieve this, I created two NodePools — one restricted to each AZ — and set the weight to 100 for both.
The idea was to let Karpenter decide freely while keeping the AZ distribution balanced.

However, when scaling up workloads, I noticed Karpenter heavily prefers eu-south-1b (around 90% of nodes), even though both NodePools have the same weight and configuration


Expected Behavior

With equal weights and similar NodePool configurations, I would expect Karpenter to roughly balance node provisioning across the two AZs over time.


Actual Behavior

When scaling a single deployment from 1 to 160 replicas (to trigger node creation), almost all new nodes are provisioned in eu-south-1b.

  • The application has no affinity rules, only a nodeSelector for arm64.
  • It's a single container per pod.
  • I’m not using TopologySpreadConstraints because many of my workloads have only one pod.

Interestingly, if I reduce the weight of the eu-south-1b NodePool to 70 while keeping eu-south-1a at 100, the resulting distribution becomes almost perfectly balanced (around 50% nodes in each AZ).

This makes me wonder:

  • Is this caused by a significant Spot capacity imbalance between AZs for ARM64 instances?
  • Or am I missing a configuration that would help achieve fair AZ balancing without manually tweaking weights?

Steps to Reproduce

  1. Create two NodePools, one restricted to eu-south-1a and one to eu-south-1b, both with:
    • weight: 100
    • capacity-type: spot
    • kubernetes.io/arch: arm64, amd64
  2. Deploy a single Deployment with:
    • replicas: 160
    • nodeSelector for arm64
  3. Observe node provisioning across AZs.

Environment

  • Karpenter version: 1.6.0
  • EKS version: 1.32
  • AWS region: eu-south-1
  • Instance types: ARM64, CPU 4 or 8, instance-generation > 2
  • Capacity type: Spot

My config nodepool

# Vedi 'README.md#nodepool' Karpenter per dettagli.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: standard-az-a
spec:
# Aggiunto weight per bilanciare con nodepool-standard-az-b
weight: 100
template:
 metadata:
   labels:
     node-scope: standard-az-a
 spec: 
   requirements:
     - key: kubernetes.io/arch
       operator: In
       values: ["arm64", "amd64"]
     - key: kubernetes.io/os
       operator: In
       values: ["linux"]
     - key: karpenter.sh/capacity-type
       operator: In
       values: ["spot"]
     - key: "karpenter.k8s.aws/instance-cpu"
       operator: In
       values: ["4", "8"]
     - key: karpenter.k8s.aws/instance-generation
       operator: Gt
       values: ["2"]
     - key: "topology.kubernetes.io/zone"
       operator: In
       values: ["eu-south-1a"]

   nodeClassRef:
     group: karpenter.k8s.aws
     kind: EC2NodeClass
     name: default
limits:
 cpu: 1000
disruption:
 consolidationPolicy: WhenEmptyOrUnderutilized
 consolidateAfter: 15s
 expireAfter: 720h

 maxSurge: "20%"
 budgets:
 - nodes: "10%"  # Limita il numero di nodi che possono essere drenati contemporaneamente
   schedule: "* * * * *"  # Cron schedule per quando applicare questo budget
   duration: "1h"  # Per quanto tempo questo budget è valido dopo l'attivazione




# Vedi 'README.md#nodepool' Karpenter per dettagli.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: standard-az-b
spec:
# Aggiunto weight per bilanciare con nodepool-standard-az-a
weight: 100
template:
 metadata:
   labels:
     node-scope: standard-az-b
 spec: 
   requirements:
     - key: kubernetes.io/arch
       operator: In
       values: ["arm64", "amd64"]
     - key: kubernetes.io/os
       operator: In
       values: ["linux"]
     - key: karpenter.sh/capacity-type
       operator: In
       values: ["spot"]
     - key: "karpenter.k8s.aws/instance-cpu"
       operator: In
       values: ["4", "8"]
     - key: karpenter.k8s.aws/instance-generation
       operator: Gt
       values: ["2"]
     - key: "topology.kubernetes.io/zone"
       operator: In
       values: ["eu-south-1b"]
   nodeClassRef:
     group: karpenter.k8s.aws
     kind: EC2NodeClass
     name: default
limits:
 cpu: 1000
disruption:
 consolidationPolicy: WhenEmptyOrUnderutilized
 consolidateAfter: 15s
 expireAfter: 720h
 maxSurge: "20%"
 budgets:
 - nodes: "10%"  # Limita il numero di nodi che possono essere drenati contemporaneamente
   schedule: "* * * * *"  # Cron schedule per quando applicare questo budget
   duration: "1h"  # Per quanto tempo questo budget è valido dopo l'attivazione

asked a year ago238 views

1 Answer
1
Accepted Answer

The imbalance you're experiencing with Karpenter provisioning nodes predominantly in eu-south-1b despite equal weights is likely due to Spot instance availability differences between the two Availability Zones.

Karpenter makes provisioning decisions based on multiple factors beyond just the NodePool weights. For Spot instances specifically, Karpenter considers the current Spot capacity availability in each AZ. If eu-south-1b consistently has better Spot availability for ARM64 instances, Karpenter will naturally prefer that zone even with equal weights.

Your observation that reducing the weight of eu-south-1b to 70 (while keeping eu-south-1a at 100) results in a balanced distribution confirms this theory. By artificially making eu-south-1b less preferred through weight adjustment, you're counterbalancing the natural Spot availability advantage that zone appears to have.

To achieve better AZ distribution without manual weight tuning, you could:

  1. Use pod topology spread constraints in your workload deployments to explicitly distribute pods across zones. This is particularly useful for deployments with multiple replicas.

  2. Consider using pod anti-affinity rules with the topology key set to "topology.kubernetes.io/zone" to encourage spreading across zones.

  3. If your workloads are machine learning distributed training jobs or other latency-sensitive applications that benefit from same-zone configurations, you might actually want to keep the current behavior rather than forcing cross-zone distribution.

  4. For critical applications where high availability is essential, you could use a deployment configuration with pod anti-affinity using "requiredDuringSchedulingIgnoredDuringExecution" to ensure pods are distributed across different nodes and zones.

The current behavior you're seeing is expected when there's an imbalance in Spot capacity availability between zones, and your approach of adjusting weights is a valid solution if you need to maintain balanced distribution across AZs.
Sources
Running highly-available applications - Amazon EKS
Cluster Autoscaler - Amazon EKS
Modernizing Snowflake Corporate’s Kubernetes Infrastructure with Bottlerocket and Karpenter | AWS Open Source Blog

answered a year ago

EXPERT

reviewed a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.