How to Configure Custom Network ENIConfig using Custom Labels in Amazon EKS

Introduction

While the official Amazon EKS VPC CNI Custom Networking documentation primarily guides users on configuring ENIConfig based on availability zones (using topology.kubernetes.io/zone), it's also possible to configure ENIConfig using custom labels or annotations.

This guide explains how to configure ENIConfig using custom labels. This approach allows for flexible configuration of ENIConfig within the same availability zone, enabling you to use different subnets based on Kubernetes labels.

Using the Custom Networking feature in Amazon EKS's VPC CNI, you can assign ENIs for pod networking to specific subnets. This guide provides detailed instructions on how to configure ENIConfig using Kubernetes labels to utilize different subnets within the same availability zone.

Step 1: Configure Custom Networking and ENIConfig

Set up Custom Networking by configuring the VPC CNI Advanced configuration as follows:

{
  "eniConfig": {
    "create": true,
    "region": "ap-northeast-2",
    "subnets": {
      "custom-workload-2a-1": {
        "id": "subnet-id",
        "securityGroups": ["sg-id"]
      },
      "custom-workload-2a-2": {
        "id": "subnet-id",
        "securityGroups": ["sg-id"]
      }
    }
  },
  "env": {
    "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG": "true",
    "ENI_CONFIG_LABEL_DEF": "custom.networking/enigroup"
  }
}

In this configuration:

• eniConfig.create: Set to true to automatically create ENIConfig resources
• eniConfig.subnets: Define different subnets within the same availability zone 2a
  • custom-workload-2a-1
  • custom-workload-2a-2
• env: Enable Custom Networking and configure ENIConfig mapping based on Label Key=custom.networking/enigroup

Step 2: Node Configuration

The custom.networking/enigroup label set in VPC CNI is a custom label, not a managed label that comes with EKS nodes by default. Therefore, nodes requiring Custom Networking need to have this label added.

For Karpenter NodePool:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
	metadata:
	  labels:
		custom.networking/enigroup: custom-workload-2a-1  # Use first subnet
		# OR
		custom.networking/enigroup: custom-workload-2a-2  # Use Second subnet

For Managed Node Groups:

# Can be applied during editon or creation in EKS Console

Managed Node Group - Edit or Create - Kubernetes labels - Add label

Important Considerations for ENIConfig:

• Node recreation is required to change ENI configuration for existing nodes
• This guide demonstrates ENIConfig configuration using custom labels. While we focus on label-based configuration, the same approach can be applied using annotations (using ENI_CONFIG_ANNOTATION_DEF).
• ENI_CONFIG_ANNOTATION_DEF takes precedence over ENI_CONFIG_LABEL_DEF
• When both settings exist, annotation-based configuration will be applied

Conclusion

• Custom Networking enables the use of different subnets within the same AZ
• Label and Annotation-based configuration provides flexible network management
• Configuration changes only apply to new nodes; existing nodes require recreation for changes to take effect

References

1. Customize the secondary network interface in Amazon EKS nodes
2. EKS Custom Networking Documentation
3. VPC CNI Plugin Documentation
4. Karpenter Documentation
5. EKS Best Practices Guide