How do I choose specific IP address subnets for Pods in my Amazon EKS cluster?

Resolution

To allocate Pod IP addresses from different subnets than your worker nodes' subnets, use custom networking.

Activate custom networking

Before you configure custom networking, review the following default behaviors:

• Nodes and Pods use IP addresses from the same CIDR ranges in your Amazon Virtual Private Cloud (VPC) cluster. To add more IP address CIDR ranges to your Amazon VPC, see How do I use multiple CIDR ranges with Amazon EKS?
• Amazon EKS assigns Pod IP addresses from the worker node's subnet.
• You can't control which subnet assigns IP addresses to Pods.
• When no IP addresses are available in a node's subnet, new Pods fail to start, even when other subnets have available addresses.
• All traffic from Pods to IP addresses outside the Amazon VPC for your Amazon EKS cluster CIDR block uses the node's main interface and IP address. The traffic uses security groups and a subnet from the node's primary elastic network interface. It doesn't use the subnet and security groups that you defined in the ENIConfig objects.
• If you use security groups for Pods, then the Pods use the security group that you specified in your SecurityGroupPolicy. The Pods don't use the security group that you specified in the ENIConfig objects.
  Note: For more information about Pod traffic routing, see Enable outbound internet access for Pods.

Activate custom networking in the Amazon VPC Container Network Interface (Amazon VPC CNI) plugin for Kubernetes. Use the Amazon VPC CNI plugin version that matches your Kubernetes version. To access the plugin, see amazon-vpc-cni-k8s on the GitHub website.

Run the following command to verify the version in your Amazon VPC for your Amazon EKS cluster:

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

If you don't have the correct plugin version, then update the Amazon VPC CNI.

Run the following command to activate custom networking:

kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

VPC CNI v1.18 and later versions support automatic subnet discovery and dynamic address allocation based on IP address use across available subnets. For more information, see Amazon VPC CNI introduces enhanced subnet discovery.

You can also use the Amazon VPC CNI plugin to do the following:

• Specify the Amazon VPC subnets to use for your Pods.
• Define separate security groups for your Pods.

Create ENIConfig objects

Before you create your ENIConfig objects, review the following default behaviors:

• Each ENIConfig object defines one subnet and a list of security groups.
• You can assign only one ENIConfig object to each node. However, you can assign the same ENIConfig object to multiple nodes.
• When you assign an ENIConfig object to a node, Pods scheduled to the node use the subnet and security groups in the ENIConfig object.
• Your ENIConfig object's name must be the name of your Availability Zone.

Run the following command to create your ENIConfig objects:

cat EOF | kubectl apply -f -  
apiVersion: crd.k8s.amazonaws.com/v1alpha1  
kind: ENIConfig  
metadata:  
  name: example-availability-zone  
spec:  
  securityGroups:   
    - example-security-group-id  
  subnet: example-subnet-id  
EOF

Note: Replace example-availability-zone with your Availability Zone. Replace example-security-group-id with your security group's ID. Replace example-subnet-id with your subnet's ID.

Assign ENIConfig objects

To assign one ENIConfig object with an Availability Zone, automatically assign ENIConfig objects to your nodes. To associate multiple ENIConfig objects with the same Availability Zone, manually assign ENIConfig objects to your nodes.

Automatically assign ENIConfig objects

Run the following command:

kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

Manually associate ENIConfig objects

Important: Make sure that the node and the subnet in the associated ENIConfig object are in the same Availability Zone.

Run the following command:

kubectl annotate node example-node-name k8s.amazonaws.com/eniConfig=example-availability-zone

Note: Replace example-node-name with your node's full identifier. Replace example-availability-zone with the name of the Availability Zone that you used when you created your ENIConfig object.

Launch new nodes

To allocate secondary network interfaces and IP addresses from the ENIConfig subnets, you must launch new nodes. Existing nodes continue to use their original networking configuration until you replace them.

Complete the following steps:

1. Check whether you use a custom Amazon Machine Image (AMI) ID with your self-managed node group managed node group.
2. If you use a custom AMI ID, then determine your maximum Pods for each node value. You must add the --cni-custom-networking-enabled parameter when you run the max-pods-calculator.sh script.
   Note: If you don't use a launch template or AMI ID, then Amazon EKS automatically sets the maximum number of Pods.
3. Update the user data script on your new nodes to use your maximum number of Pods based on your operating system (OS):
   Amazon Linux 2, Bottlerocket, Ubuntu 20.04 and later, Windows Server 2019 and later

   #!/bin/bash  
   /etc/eks/bootstrap.sh example-cluster-name --use-max-pods false --kubelet-extra-args '--max-pods=example-max-pods'

   --user-data '#!/bin/bash /etc/eks/nodeadm init \ --container-runtime containerd \ --cluster-name example-cluster-name \ --max-pods example-max-pods'

   Note: In the previous commands, replace example-cluster-name with your EKS cluster's name. Replace example-max-pods with your max Pods per node value.
   For more information about bootstrap, see awslabs/amazon-eks-ami on the GitHub website. For information about nodeadm, see Upgrade from Amazon Linux 2 to Amazon Linux 2023.
   Amazon Linux 2023 (AL2023)
4. Recreate your Pods to use your new custom networking configuration.