Karpenter Scheduling Nodes into Subnet with No Available Private IPs

EKS Automode has networking capability that manages node and pod networking. So it uses AWS VPC CNI as network plugin by default. In Automode, not every configuration parameter of aws vpc cni is supported. It can be controlled through EKS Node Class that will be used by karpenter while placing nodes,pods. There will be a default nodeclass available with automode.

So, only way to control networking components is through EKS Node class in this case and it is through "specsubnetselectorterms" which exposes only tags and IDs. But by default, eks auto mode handles node placement based on the available IPV4 address of the subnet. So, there is not much control exposed to clients beyond that.

Besides, i would like to know few things regarding your error which follows

1. EKS Automode handles Prefix delegation while creating primary network interface of a node. So it provisions a /28 IPV4 prefixes to the primary ENI. So, by default there should be 14 IPV4 addresses available for aws vpc cni to allocate to a pod/workload. The only case i can think of a failure here is that you have more than 14 daemons to run on node.
2. If in a case, that there are no available IPV4 addresses at all, how does karpenter places a node in that subnet as VPC CNI also uses the same aws subnet for ip address allocation

Hi Rahib,

Thanks for sharing the detailed subnet information. Looking at your subnet configuration, I can see a critical issue with your current setup:

| AvailableIPs |     CIDR       |            ID              |
+--------------+----------------+----------------------------+
|  215         |  10.0.23.0/24  |  subnet-1                  |
|  105         |  10.0.22.0/25  |  subnet-2                  |
|  0           |  10.0.21.0/26  |  subnet-3                  |

Critical Issue: Unbalanced Subnet Sizes

Your subnets have very different capacities due to different subnet masks:

• subnet-1 (/24): ~251 usable IPs
• subnet-2 (/25): ~123 usable IPs
• subnet-3 (/26): ~59 usable IPs

These are relatively small subnets for a Kubernetes environment, and the size imbalance means that even if Karpenter balances pod placement across AZs, the smaller subnets will exhaust their IPs much faster than the larger ones.

Important: If you want to implement balanced pod distribution across AZs, you need to ensure your subnets have sufficient and balanced IP capacity. Otherwise, you'll just shift the IP exhaustion problem to different subnets at different times, creating unbalanced free IP distribution and operational complexity.

Immediate Solutions for IP Exhaustion

1. Expand Your IP Address Space You have a couple of options to add more IPs to your environment:

• Create new subnets using any unused IP space within your current VPC CIDR
• Add a secondary CIDR to your VPC and create larger, uniformly-sized subnets from that range

Both approaches will require you to review any hardcoded CIDR references in your applications or infrastructure.

2. Safe Implementation Strategy The safest way to implement new subnets is by creating new NodePools that target the new subnets specifically. This approach offers several benefits:

• Your existing workloads continue running unchanged on current subnets
• You can gradually migrate workloads using node selectors and affinity rules
• New NodePools scale independently based on demand
• You get better subnet isolation and network segmentation

Add Observability to Prevent Future Issues

To avoid this problem recurring, consider monitoring these key metrics:

• Remaining IPs per subnet / Used IPs / Number of pods
• Per-node IP utilization to see how many IPs each worker reserves vs. actually uses

If you discover IP waste at the node level, you can optimize the VPC CNI configuration by adjusting WARM ENI and IP settings to balance quick pod scaling with IP efficiency. Check the EKS VPC CNI best practices guide for configuration examples.

Application-Level Optimizations

You can use Topology Spread Constraints in your deployments to distribute pods more evenly:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule

Critical Consideration: With your current uneven subnet sizes, balanced pod distribution will cause smaller subnets to run out of IPs first, leading to unbalanced free IP availability across AZs. For balanced distribution to work effectively, you need subnets with equal or proportional capacity.

Recommended Action Plan

Given that you have 4 pending pods all scheduled on the same node, I'd recommend:

1. Immediately: Create new, larger subnets with uniform sizing (e.g., all /22 or /21 to provide ample room)
2. Short-term: Set up new NodePools targeting subnets with available capacity
3. Long-term: Implement monitoring and gradually migrate workloads to the new, properly-sized and balanced subnets

The combination of small subnet sizes and imbalanced capacity is the root cause of your IP exhaustion issue. Simply balancing pods without addressing the underlying capacity imbalance will only create different operational challenges.