What are the best practices to configure the Amazon VPC CNI plugin to use an IP address in VPC subnets with Amazon EKS?

Short description

One of the primary components of the Amazon VPC CNI is the L-IPAM daemon. You configure this daemon to allocate IP addresses to nodes.

If a new pod is scheduled on a node, then the kubelet invokes the CNI binary. The CNI binary calls the L-IPAMD to get an IP address for the new pod. The pod then tracks the elastic network interfaces and IP addresses that are attached to the instance.

You can use certain configuration variables to control how many network interfaces and IP addresses are maintained. For more information, see WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET and WARM_PREFIX_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET (from the GitHub website).

Resolution

The following are basic best practices for using configuration variables to control how many network interfaces and IP addresses are maintained.

WARM_ENI_TARGET

Use the WARM_ENI_TARGET variable to determine how many elastic network interfaces the L-IPAMD keeps available so that pods are immediately assigned an IP address when scheduled on a node.

• To prevent depleting available IP addresses in the subnet, make sure to check the worker node instance type and the maximum number of network interfaces and private IPv4 addresses per interface. For example, if you set WARM_ENI_TARGET=3 for an m5.xlarge node, then three elastic network interfaces are always attached to the node. The node then assigns 45 IP addresses, 15 per elastic network interface. Because the 45 IP addresses are reserved for this node, those addresses can't be used for pods that are scheduled on other worker nodes.
• If you expect your application to scale greatly, then you can use WARM_ENI_TARGET to quickly accommodate newly scheduled pods.

WARM_IP_TARGET

Use the WARM_IP_TARGET variable to make sure that you always have a defined number of IP addresses available in the L-IPAMD's warm pool.

• For clusters with low productivity, use WARM_IP_TARGET so that only the required number of IP addresses are assigned to the network interface. This prevents the IP addresses of the elastic network interfaces from being blocked.

MINIMUM_IP_TARGET

Use the MINIMUM_IP_TARGET to make sure that a minimum number of IP addresses are assigned to a node when it initially appears. This variable is generally used with the WARM_IP_TARGET variable.

• If you know the minimum number of pods that you'll run per node, then use MINIMUM_IP_TARGET so that the required number of IP addresses are assigned. If the IP addresses are readily available, then the pods can receive them as the pods are scheduled.
• Set this variable with WARM_IP_TARGET to make sure that there are available IP addresses on the node for future pods.

WARM_PREFIX_TARGET

Use the WARM_PREFIX_TARGET variable to make sure that you always have a defined number of prefixes (/28 CIDR blocks) added to the instance's network interface. You can use WARM_PREFIX_TARGET only for CNI version 1.9.0 or later, and you must activate the Amazon VPC CNI IP address prefix assignment capability.

• If you use the IP address prefix assignment, then make sure that the WARM_PREFIX_TARGET variable is set to a value greater than or equal to 1. If it's set to 0, then you receive the following error:

Error: Setting WARM_PREFIX_TARGET = 0 is not supported while WARM_IP_TARGET/MINIMUM_IP_TARGET is not set. 
Please configure either one of the WARM_{PREFIX/IP}_TARGET or MINIMUM_IP_TARGET env variables

• For smaller subnets, use WARM_IP_TARGET with WARM_PREFIX_TARGET to avoid allocating too many prefixes. Allocating too many prefixes can deplete available IP addresses.

To learn more about how these configuration variables affect IP address utilization, see CNI configuration variables (from the GitHub website).

---