How do I resolve and monitor insufficient IP addresses in my Amazon EKS clusters?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Use enhanced subnet discovery in the Amazon VPC CNI

Create the Amazon Virtual Private Cloud (VPC) Container Network Interface (CNI) plug-in that uses enhanced subnet discovery to allocate IP addresses from subnets in the same VPC. The plug-in also balances loads across Availability Zones.

Note: You must use Amazon VPC CNI add-on version 1.18.0 or later. Also, your clusters must run Amazon EKS 1.25 or later.

To check whether enhanced subnet discovery is activated, run the following command:

kubectl describe ds aws-node -n kube-system | grep ENABLE_SUBNET_DISCOVERY

To turn on enhanced subnet discovery, run one of the following commands for the tool that you use.

kubectl:

kubectl set env daemonset aws-node -n kube-system ENABLE_SUBNET_DISCOVERY=true -c aws-node

AWS CLI:

aws eks update-addon --cluster-name CLUSTER_NAME --region AWS_REGION \
--addon-name vpc-cni \
--configuration-values '{"env":{"ENABLE_SUBNET_DISCOVERY":"true"}}'

Note: Replace CLUSTER_NAME with your cluster name and AWS_REGION with your AWS Region.

Add the kubernetes.io/role/cni tag to your subnets.

Use custom networking

For security purposes, sometimes Pods must run on separate networks with different security groups. Use custom networking to assign Pod IP addresses from secondary CIDR ranges in your VPC.

When you use custom networking in the Amazon VPC CNI, custom networking creates secondary network interfaces in the subnet in the ENIConfig file. The Amazon VPC CNI then assigns Pod IP addresses from the subnet so that your Pods can use different security groups from the node's primary network interface.

Note: If you use both enhanced subnet discovery and custom networking, then custom networking takes precedence.

For the secondary CIDR range, use CIDRs (/16) from CG-NAT. For example, use 100.64.0.0/10 or 198.19.0.0/16. These ranges are less common in corporate networks than RFC1918 ranges.

For more information, see Customize the secondary network interface in Amazon EKS nodes.

Activate prefix delegation

Nitro-based Amazon Elastic Compute Cloud (EC2) instances can use prefix delegation. Prefix delegation assigns /28 IPv4 address prefixes instead of single IP addresses.

To turn on prefix delegation, run the one of the following commands for your operating system (OS).

Linux:

kubectl set env ds aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Windows:

kubectl patch configmap amazon-vpc-cni -n kube-system --type merge -p '{"data": {"enable-windows-prefix-delegation": "true", "enable-windows-ipam": "true"}}'

When your subnet's IP addresses become fragmented, then prefix delegation might fail because it requires contiguous blocks of IP addresses. Then, you might receive the following error message in the VPC CNI logs:

"Failed to allocate a private IP/Prefix address: InsufficientCidrBlocks: There are not enough free cidr blocks in the specified subnet to satisfy the request."

To resolve this issue, use VPC subnet CIDR reservations to reserve IP addresses for prefix allocation. The VPC CNI plug-in then allocates prefixes from the reserved IP addresses.

Set CNI optimization parameters

To keep a specific number of IP addresses available, set the WARM_IP_TARGET, MINIMUM_IP_TARGET, and WARM_ENI_TARGET parameters.

Set the WARM_IP_TARGET parameter to keep a specified number of IP addresses on each node. Set the MINIMUM_IP_TARGET parameter to always keep a minimum number of IP addresses available. Set the WARM_ENI_TARGET parameter to keep a specified number of network interfaces that have available IP addresses.

For more information see, WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET on the GitHub website.

To set WARM_IP_TARGET, run the following command:

kubectl set env ds aws-node -n kube-system WARM_IP_TARGET=YOUR_TARGET_VALUE

Note: Replace YOUR_TARGET_VALUE with your warm IP target number.

To set MINIMUM_IP_TARGET, run the following command:

kubectl set env ds aws-node -n kube-system MINIMUM_IP_TARGET=YOUR_MIN_VALUE

Note: Replace YOUR_MIN_VALUE with your minimum IP target number.

To set WARM_ENI_TARGET, run the following command:

kubectl set env ds aws-node -n kube-system WARM_ENI_TARGET=YOUR_ENI_VALUE

Note: Replace YOUR_ENI_VALUE with your warm network interface target number.

Monitor your IP address usage

To monitor your cluster's IP address usage to prevent address depletion, take the following actions:

• Install the CNI metrics helper to monitor IP address allocation. To install the tool, see cni-metrics-helper on the GitHub website. For more information about the tool see, CNI metrics helper.
• Use Amazon Cloudwatch Container Insights to monitor the following metrics for your cluster's IP address usage:
  awscni_total_ip_addresses: The total number of IP addresses that the Amazon VPC CNI manages. 
  awscni_assigned_ip_addresses: The number of IP addresses that you assigned to Pods. 
  awscni_available_ip_addresses: The number of available IP addresses for new Pods.
• Create a CloudWatch alarm for when the available IP addresses are below your required minimum. Also, create an alarm for when your usage exceeds your threshold.

Implement best practices for new cluster deployments

For new Amazon EKS cluster deployments, implement the following best practices:

• When you create your VPC, configure CIDR blocks with sufficient IP addresses. For example, /16 provides 65,546 IP addresses. For more information, see How Amazon VPC works.
• Assign an IPv6 address to your cluster.