How can I tag the Amazon VPC subnets in my Amazon EKS cluster for automatic subnet discovery by load balancers or ingress controllers?

Short description

The Kubernetes Cloud Controller Manager (cloud-controller-manager) and AWS Load Balancer Controller (aws-load-balancer-controller) query a cluster's subnets to identify them. This query uses the following tag as a filter:

kubernetes.io/cluster/cluster-name

Note: Replace cluster-name with your Amazon EKS cluster's name.

The Cloud Controller Manager and AWS Load Balancer Controller both require subnets to have either of the following tags:

kubernetes.io/role/elb

-or-

kubernetes.io/role/internal-elb

Note: If you don't use the preceding tags, then Cloud Controller Manager determines if a subnet is public or private by examining its associated route table. Unlike private subnets, public subnets use an internet gateway to get a direct route to the internet.

If you don't associate your subnets with either tag and you're using AWS Load Balancer Controller, then you receive an error.

For example, if you're troubleshooting the Kubernetes service and you run the kubectl describe service your-service-name command, then you receive the following error:

Events:
  Type     Reason                      Age               From                Message
  ----     ------                      ----              ----                -------
  Normal   EnsuringLoadBalancer        9s (x2 over 14s)  service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  9s (x2 over 14s)  service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service default/guestbook: could not find any suitable subnets for creating the ELB

If you're troubleshooting the Application Load Balancer Ingress Controller and you run the kubectl logs your-aws-load-balancer-controller-pod-name command, then you receive the following error:

E0121 22:44:02.864753       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to
retrieval of subnets failed to resolve 2 qualified subnets. Subnets must contain the kubernetes.io/cluster/\u003ccluster name\u003e tag with a value of shared or owned and the kubernetes.io/role/elb tag signifying it should be used for ALBs Additionally, there must be at least 2 subnets with unique availability zones as required by ALBs. Either tag subnets to meet this requirement or use the subnets annotation on the ingress resource to explicitly call out what subnets to use for ALB creation. The subnets that did resolve were []"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"2048-ingress"}

Note: If you create the VPC using eksctl, then all the subnets in that VPC have the kubernetes.io/role/elb and kubernetes.io/role/internal-elb tags.

Resolution

Choose the appropriate option for tagging your subnets:

For public and private subnets used by load balancer resources

Tag all public and private subnets that your cluster uses for load balancer resources with the following key-value pair:

Key: kubernetes.io/cluster/cluster-name
Value: shared

Note: Replace cluster-name with your Amazon EKS cluster's name. The shared value allows more than one cluster to use the subnet.

For private subnets used by internal load balancers

To allow Kubernetes to use your private subnets for internal load balancers, tag all private subnets in your VPC with the following key-value pair:

Key: kubernetes.io/role/internal-elb
Value: 1

For public subnets used by external load balancers

To allow Kubernetes to use only tagged subnets for external load balancers, tag all public subnets in your VPC with the following key-value pair:

Key: kubernetes.io/role/elb
Value: 1

Note: Use the preceding tag instead of using a public subnet in each Availability Zone.

Related information

Amazon EKS VPC and subnet requirements and considerations

Subnet Auto Discovery on the GitHub website

Application load balancing on Amazon EKS