How do I automatically discover the subnets that my Application Load Balancer uses in Amazon EKS?

6 minute read

I want to automatically discover the subnets that my Application Load Balancer uses in Amazon Elastic Kubernetes Service (Amazon EKS).

Short description

To identify the subnet that your Application Load Balancer uses, the aws-load-balancer-controller AWS Load Balancer Controller queries the cluster's subnets. The query uses the following tags as a filter:

kubernetes.io/cluster/cluster-name      shared
kubernetes.io/role/elb                    1
OR    
kubernetes.io/role/internal-elb           1

Note: Replace cluster-name with your Amazon EKS cluster's name.

To allow the AWS Load Balancer Controller to automatically discover the subnets that your Application Load Balancer uses, tag your subnets.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Add tags to subnets

Complete the following steps:

Deploy the AWS Load Balancer Controller on your Amazon EKS cluster.
To verify that you correctly installed the AWS Load Balancer Controller, run the following command:
```
kubectl get deployment -n kube-system aws-load-balancer-controller
```
Note: If you deployed the AWS Load Balancer Controller in a different namespace, then replace -n kube-system with the correct namespace.
Create a Kubernetes Ingress resource on your cluster with the following annotation:
```
annotations:    kubernetes.io/ingress.class: alb
```
Note: The AWS Load Balancer Controller helps create load balancers. The Ingress resource configures the Application Load Balancer to route HTTP and HTTPS traffic to different Pods in your cluster.
Under the Ingress object annotations, enter internal to create an internal load balancer or internet-facing to create a public load balancer:
```
alb.ingress.kubernetes.io/scheme: internal            
```
-or-
```
alb.ingress.kubernetes.io/scheme: internet-facing
```
Note: You can use tags for subnet automatic discovery instead of the manual alb.ingress.kubernetes.io/subnets annotation.
For cluster versions 1.18 and earlier, Amazon EKS adds the following tag to all subnets that Amazon EKS passes during cluster creation:
```
kubernetes.io/cluster/CLUSTER_NAME
```
Note: Replace CLUSTER_NAME with your cluster name.
The kubernetes.io/cluster/CLUSTER_NAME tag is only required if you use AWS Load Balancer Controller version 2.1.1 or earlier. Amazon EKS doesn't automatically add or remove this tag. Although the tag was required for clusters before version 1.19, it remains on subnets even after cluster upgrades.

To verify that your subnets have the correct tags, run the following describe-subnets AWS CLI command:

aws ec2 describe-subnets --subnet-ids your-subnet-abcdefghxyz

Note: Replace your-subnet-abcdefghxyz with your subnet ID.

To deploy a sample application, run the following command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/2048/2048_full.yaml

Note: The preceding command creates an Application Load Balancer because of the Ingress object.

To verify that Amazon EKS created the Ingress resource and an associated Application Load Balancer, run the following command:

kubectl get ingress/2048-ingress -n game-2048

Troubleshoot tag errors

Take the following actions based on the error that you receive when you use tags to automatically discover subnets.

Permissions denied issues

Your AWS account's AWS Identity and Access Management (IAM) role for the AWS Load Balancer Controller must have the required permissions. If it doesn't, then you receive the following error message:

"{"level":"error","ts":1621443417.9175518,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":" ingress-2048","namespace":" game-2048","error":"couldn't auto-discover subnets: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 72ee57ae-f804-4f81-b069-8b04114b67b0"}"

To resolve this issue, complete the following steps:

To verify that your service account is associated with the AWS Load Balancer Controller, run the following command:

kubectl get deploy aws-load-balancer-controller -n kube-system -o yaml | grep -i serviceAccount

Note: If you deployed the AWS Load Balancer Controller in a different namespace, then replace -n kube-system with the correct namespace.
Example output:

serviceAccount: aws-load-balancer-controller
serviceAccountName: aws-load-balancer-controller

To identify the IAM role that's attached to the service account that you associated with the AWS Load Balancer Controller, run the following command:

kubectl describe sa aws-load-balancer-controller -n kube-system | grep role-arn

Example output:

annotations: eks.amazonaws.com/role-arn: arn:aws:iam::abcdefxyz:role/eksctl-cluster18-addon-iamserviceaccount-kub-Role1-abcdefxyz

Note: If you don't use IAM roles for service accounts (IRSA), then make sure that the Amazon EKS worker node IAM role has the required permissions. To view the required permissions, see iam_policy.json on the GitHub website.

Single subnet discovery issue

You receive one of the following error messages when your AWS Load Balancer Controller doesn't discover at least one subnet:

"{"level":"error","ts":1608229710.3212903,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"ingress-2048","namespace":"game-2048","error":"couldn't auto-discover subnets: unable to resolve at least one subnet"}"

-or-

"kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to
retrieval of subnets failed to resolve 2 qualified subnets. Subnets must contain the kubernetes.io/cluster/\u003ccluster name\u003e tag with a value of shared or owned and the kubernetes.io/role/elb tag signifying it should be used for ALBs Additionally, there must be at least 2 subnets with unique availability zones as required by ALBs. Either tag subnets to meet this requirement or use the subnets annotation on the ingress resource to explicitly call out what subnets to use for ALB creation. The subnets that did resolve were []" "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"2048-ingress"}"

To resolve this issue, add tags to your subnets to allow the AWS Load Balancer Controller to use auto-discovery to create a load balancer.

For private subnets, add the following tag:

kubernetes.io/role/internal-elb                1

For public subnets, add the following tag:

kubernetes.io/role/elb                         1

Note: You can manually assign subnets to your load balancer with the alb.ingress.kubernetes.io/subnets annotation. To view the required permissions, see iam_policy.json on the GitHub website.

Tag your subnets with the following format without any leading or trailing spaces:

For Key, enter kubernetes.io/cluster/your-cluster-name.
For Value, enter shared or owned.

If you use the AWS Load Balancer Controller version 2.1.1 or earlier, then you must tag your subnets in the preceding format. Tagging is optional for versions 2.1.2 or later.

It's a best practice to tag a subnet in the following scenarios:

You have multiple clusters that run in the same virtual private cloud (VPC).
You have multiple AWS services that share subnets in a VPC.
You want more control over where load balancers are allocated for each cluster.

Multiple subnet discovery errors

You receive the following error message when your AWS Load Balancer Controller doesn't discover two or more qualified subnets:

"{"level":"error","ts":"2024-08-12T19:01:27Z","msg":"Reconciler error","controller":"ingress","object":{"name":"ingress-2048","namespace":"game-2048"},"namespace":"game-2048","name":"ingress-2048","reconcileID":"1234567","error":"couldn't auto-discover subnets: unable to resolve at least one subnet (2 match VPC and tags: [kubernetes.io/role/internal-elb], 2 have fewer than 8 free IPs)"}"