- Newest
- Most votes
- Most comments
The imbalance you're experiencing with Karpenter provisioning nodes predominantly in eu-south-1b despite equal weights is likely due to Spot instance availability differences between the two Availability Zones.
Karpenter makes provisioning decisions based on multiple factors beyond just the NodePool weights. For Spot instances specifically, Karpenter considers the current Spot capacity availability in each AZ. If eu-south-1b consistently has better Spot availability for ARM64 instances, Karpenter will naturally prefer that zone even with equal weights.
Your observation that reducing the weight of eu-south-1b to 70 (while keeping eu-south-1a at 100) results in a balanced distribution confirms this theory. By artificially making eu-south-1b less preferred through weight adjustment, you're counterbalancing the natural Spot availability advantage that zone appears to have.
To achieve better AZ distribution without manual weight tuning, you could:
-
Use pod topology spread constraints in your workload deployments to explicitly distribute pods across zones. This is particularly useful for deployments with multiple replicas.
-
Consider using pod anti-affinity rules with the topology key set to "topology.kubernetes.io/zone" to encourage spreading across zones.
-
If your workloads are machine learning distributed training jobs or other latency-sensitive applications that benefit from same-zone configurations, you might actually want to keep the current behavior rather than forcing cross-zone distribution.
-
For critical applications where high availability is essential, you could use a deployment configuration with pod anti-affinity using "requiredDuringSchedulingIgnoredDuringExecution" to ensure pods are distributed across different nodes and zones.
The current behavior you're seeing is expected when there's an imbalance in Spot capacity availability between zones, and your approach of adjusting weights is a valid solution if you need to maintain balanced distribution across AZs.
Sources
Running highly-available applications - Amazon EKS
Cluster Autoscaler - Amazon EKS
Modernizing Snowflake Corporate’s Kubernetes Infrastructure with Bottlerocket and Karpenter | AWS Open Source Blog
Relevant content
asked a year ago
asked 3 years ago
asked a year ago
