Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Diagnosing Subnet IP Fragmentation for EKS Prefix Delegation

8 minute read
Content level: Advanced
2

Amazon EKS prefix delegation and EKS Auto Mode assign /28 prefixes (16 contiguous addresses) to worker nodes instead of individual IP addresses. A subnet can report plenty of free addresses and still fail to attach a prefix. This article explains why that happens, how to diagnose it with an open-source read-only tool, and how to remediate it.

Background

Prefix delegation assigns a /28 (16 addresses) to a network interface instead of one address at a time. EC2 can only carve a /28 from 16 contiguous addresses that are all free, and the block must not overlap any address or prefix already assigned to a resource. DescribeSubnets reports a count of free addresses, not a count of free blocks, so a subnet with hundreds of free addresses can have zero free /28 blocks when those addresses are scattered.

When the VPC CNI cannot find a free /28, it logs this (see Prefix Mode for Linux in the EKS Best Practices Guide):

failed to allocate a private IP/Prefix address: InsufficientCidrBlocks: There are not enough free cidr blocks in the specified subnet to satisfy the request.

What makes a /28 block usable

A /28 prefix must align to a 16-address boundary, for example 10.0.0.0/28 and 10.0.0.16/28. EC2 rejects a prefix if any address in the block is already assigned or reserved (see Prefix delegation for EC2 network interfaces).

AWS reserves 5 addresses in every subnet: the first four (network address, VPC router, DNS, and one reserved for future use) and the last (broadcast) (see Subnet sizing for IPv4). Because those sit at the extremes of the range, the first and last /28 of any subnet always contain a reserved address and can never hold a prefix. That rules out the smallest subnets completely: a /28 is a single /28 block holding all five reserved addresses, and a /27 is only two /28 blocks, the first with the four low addresses and the second with the broadcast, so neither is ever free. A /26 is the smallest subnet that can hold a usable /28 block, because it splits into four /28 blocks and the two interior ones sit clear of the reserved addresses; every larger subnet just adds more interior blocks.

Why fragmentation occurs

Every VPC resource that needs a private address creates an ENI: EC2 instances, EKS pods, Lambda functions, NAT gateways, load balancers, RDS, and VPC endpoints. As these come and go, their addresses spread across the subnet's /28 blocks and leave partially used blocks behind. Fragmentation tends to build up with:

  • Mixed workloads (Lambda, RDS, EKS) sharing one subnet.
  • Long-running clusters with frequent node scaling.
  • Turning on prefix delegation in a subnet already fragmented by individual IP allocation.
  • Orphaned ENIs from deleted resources that still hold addresses.

Diagnosing with a read-only tool

Subnet Fragmentation Analyzer is an open-source Python tool that reads the EC2 and EKS APIs, maps every in-use address to its /28 block, and shows which blocks are free, fragmented, full, reserved, or already carrying a prefix.

Point it at a cluster with --cluster and it discovers every subnet the cluster touches (control plane, managed node groups, Fargate profiles, and the cluster, Karpenter, and CNI subnet tags), labels each subnet with where it was found, flags EKS Auto Mode, then analyzes each subnet in turn:

$ python3 subnet_frag.py --cluster prod-eks --region us-east-1

══════════════════════════════════════════════════════════════════════
  Cluster: prod-eks (us-east-1)
  VPC:     vpc-0a1b2c3d4e5f6a7b
══════════════════════════════════════════════════════════════════════

  Discovered 4 subnet(s):
    subnet-0a1b2c3d4e5f6aaaa  [control-plane]
    subnet-0a1b2c3d4e5f6bbbb  [control-plane]
    subnet-0a1b2c3d4e5f6cccc  [nodegroup:ng-general, tag:cluster, tag:cni-role]
    subnet-0a1b2c3d4e5f6dddd  [nodegroup:ng-general, tag:cluster, tag:cni-role]

══════════════════════════════════════════════════════════════════════
  Subnet: prod-eks-pods-1a (subnet-0a1b2c3d4e5f6cccc)
  CIDR:   10.0.0.0/23  AZ: us-east-1a
══════════════════════════════════════════════════════════════════════

  CIDR Reservations:
    10.0.1.128/28 (prefix) reserved-for-pods

  IPs:  53 used | 358 free | 5 AWS-reserved
  /28 blocks: 32 total | 4 free | 18 fragmented | 2 full | 2 reserved | 6 prefix-allocated
  Fragmentation: [████████████████░░░░] 84% (HIGH)

  Owner Type             ENIs    IPs
  ────────────────────────────────────
  eks pod                  14     28
  ec2 primary               8      8
  elb                       3      6
  lambda                    5      5
  rds                       2      4
  nat gateway               1      1
  vpc endpoint              1      1

  /28 Block Map:  ■ full  ▣ fragmented  □ free  R reserved  A prefix-allocated  P prefix-reserved
  R ▣ ▣ A A ▣ ▣ ■ ▣ □ ▣ A A ▣ ▣ ▣
  ▣ □ ▣ A A ▣ ■ ▣ P ▣ ▣ □ ▣ ▣ ▣ R

  Prefix-allocated blocks (6):
    10.0.0.48/28  → eks_pod (eni-0a1b2c3d4e5f60002)
    10.0.0.64/28  → eks_pod (eni-0a1b2c3d4e5f60002)
    10.0.0.176/28 → eks_pod (eni-0a1b2c3d4e5f60007)
    10.0.0.192/28 → eks_pod (eni-0a1b2c3d4e5f60007)
    10.0.1.48/28  → eks_pod (eni-0a1b2c3d4e5f6000b)
    10.0.1.64/28  → eks_pod (eni-0a1b2c3d4e5f6000b)

  (...)

The discovery header lists the subnets the cluster touches and how each one was found. The two control-plane subnets carry the EKS-managed interfaces; the two pod subnets, found through the node group and the kubernetes.io/cluster and kubernetes.io/role/cni tags, are where prefix delegation pressure builds, so they are the subnets to focus on. A subnet labeled only tag:cni-role was matched by the cluster-agnostic pod-subnet tag alone, so in a shared VPC it may belong to another cluster. For EKS Auto Mode clusters there are no node groups or Fargate profiles, so the data plane is found through the CNI tag and the header shows [EKS Auto Mode]. Add --dry-run to list the discovered subnets and exit without analyzing them.

To analyze a single subnet directly, pass --subnet-id subnet-0abc123def4567890 --region us-east-1. --list-enis adds a full ENI inventory with owner attribution, --node-recs ranks nodes by how many blocks draining them would recover, and --json emits structured output for automation. The repository documents the full command set, output, and required permissions.

Analyzing the output

The fragmentation score answers one question: excluding the blocks that can never hold a prefix (AWS-reserved) or already carry one (prefix-allocated), what fraction of the rest cannot currently take a new prefix? A HEALTHY subnet has plenty of free blocks. A HIGH or CRITICAL subnet is at or near the point where the next node or pod scale-up fails with InsufficientCidrBlocks.

The block map shows where the addresses sit, one square per /28 block:

  • □ free: available for a new prefix.
  • ▣ fragmented: at least one address in use, so the whole /28 is unusable for a prefix even though it still has free addresses. This is the fragmentation that a DescribeSubnets free-IP count hides.
  • ■ full: every address in use.
  • R reserved: contains one of the AWS-reserved addresses, so it can never hold a prefix and is excluded from the score.
  • A prefix-allocated: a /28 already delegated to an ENI. These are prefix delegation working as intended, and they do not count against the score.
  • P prefix-reserved: a free block that sits inside a CIDR reservation set aside for prefixes. It is still a free candidate and counts in the score exactly like □.

In the example above, the six A blocks show prefix delegation already succeeding: six /28 prefixes are attached and serving pods. The risk is the next allocation. Only four free blocks remain against eighteen fragmented ones, so the subnet is close to failing even though it still reports 358 free addresses. That is the 84% (HIGH) score.

The owner breakdown, and --list-enis for the per-interface detail, tell you who is holding the scattered addresses, and --node-recs ranks the nodes worth draining. What you see points to a remediation in the next section:

  • Detached (available) or orphaned ENIs, and addresses on stopped instances, are space you can reclaim right away.
  • A fragmented block whose addresses all belong to a single node is a clean drain candidate.
  • Fragmentation that keeps returning after cleanup needs a structural fix, not another cleanup pass.

See Remediation for how to apply each one.

Remediation

Warning: draining a node evicts and reschedules the pods running on it, which interrupts those workloads. Cordon and drain during a maintenance window, after confirming the pods can reschedule elsewhere.

  • Delete orphaned ENIs. Interfaces in the available state are detached and still hold addresses.
  • Terminate stopped instances. A stopped instance keeps its ENIs and their addresses.
  • Drain and replace nodes. This releases secondary ENIs, and replacement nodes tend to consolidate into fewer blocks. Use Pod Disruption Budgets to protect critical workloads while draining.
  • Reserve space with a subnet CIDR reservation of type prefix. It sets aside a range for prefixes only, and the VPC CNI then allocates prefixes from the reserved space (see Subnet CIDR reservations).
  • Use dedicated pod subnets. With EKS custom networking, pod ENIs use separate subnets; keeping those subnets free of other resources prevents cross-service fragmentation.

Get the tool

Subnet Fragmentation Analyzer is read-only and needs only EC2 and EKS describe permissions. Source and full documentation: github.com/awslabs/aws-support-tools/tree/master/EKS/subnet-fragmentation.

References

No comments

Relevant content