Why can't my AWS Load Balancer Controller find my subnet in Amazon EKS?

Short description

If your AWS Load Balancer Controller can't find your subnet in Amazon EKS, then you receive an error. To troubleshoot your error, complete the relevant steps for the error message that you receive.

The following error occurs when your account's AWS Identity and Access Management (IAM) role for the AWS Load Balancer Controller doesn't have the required permissions:

{"level":"error","ts":1621443417.9175518,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":" ingress-2048","namespace":" game-2048","error":"couldn't auto-discover subnets: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 72ee57ae-f804-4f81-b069-8b04114b67b0"}

To resolve this error, complete the steps in the Resolve the permission denied error section.

-or-

The following error occurs when your AWS Load Balancer Controller can't discover at least one subnet:

{"level":"error","ts":1608229710.3212903,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"ingress-2048","namespace":"game-2048","error":"couldn't auto-discover subnets: unable to resolve at least one subnet"}

To resolve this error, complete the steps in the Resolve the single subnet discovery error section.

-or-

The following errors occur when your AWS Load Balancer Controller can't discover two or more qualified subnets:

"msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to resolve 2 qualified subnet with at least 8 free IP Addresses for ALB

{"level":"error","ts":1606329481.2930484,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"reciter-ing","namespace":"reciter","error":"InvalidSubnet: Not enough IP space available in subnet-xxxxxxxxxxxxxx. ELB requires at least 8 free IP addresses in each subnet.\n\tstatus code: 400, request id: 2a37780c-f411-xxxxx-xxxxx-xxxxxxxxx"}

To resolve these errors, complete the steps in the Resolve multiple subnet discovery errors section.

Resolution

Resolve the permission denied error

1. To verify that your service account is associated with the AWS Load Balancer Controller, run the following command:

   $ kubectl get deploy aws-load-balancer-controller -n kube-system -o yaml | grep -i serviceAccount

   You receive an output similar to the following message:

   serviceAccount: aws-load-balancer-controller
   serviceAccountName: aws-load-balancer-controller

   Note: If your deployment is deployed in a different namespace, then replace -n kube-system with the appropriate namespace.

2. Verify which IAM role is attached to the service account that's associated with the AWS Load Balancer Controller:

   $ kubectl describe sa aws-load-balancer-controller -n kube-system | grep role-arn

   You receive an output similar to the following message:

   annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/eksctl-cluster18-addon-iamserviceaccount-kub-Role1-xxxxxxxxxxxxx

3. Grant all relevant permissions (from the AWS Load Balancer Controller GitHub page) to the IAM role that you identify in step 2, such as ec2:DescribeAvailabilityZones. For more information on how AWS Load Balancer Controller assumes an IAM role to perform API calls, see IAM roles for service accounts.

Resolve the single subnet discovery error

Allow the AWS Load Balancer Ingress Controller to use auto-discovery to create a load balancer. To do this, add the appropriate tags on your subnets:

Private subnets tags

kubernetes.io/role/internal-elb                Set to 1 or empty tag value for internal load balancers

Public subnets tags

kubernetes.io/role/elb                         Set to 1 or empty tag value for internet-facing load balancers

Note: You can manually assign subnets to your load balancer with the alb.ingress.kubernetes.io/subnets annotation.

The following example is a subnet with the correct tags for a cluster with an internal load balancer (private subnet):

kubernetes.io/role/internal-elb          1

The following example is a subnet with the correct tags for a cluster with a public load balancer (public subnet):

kubernetes.io/role/elb

Tag your subnets with the appropriate format:
Key: kubernetes.io/cluster/your-cluster-name
Value: shared or owned

Important: If you're using the AWS Load Balancer Controller version v2.1.1 or earlier, then you must tag your subnets in this format. Tagging is optional for versions 2.1.2 or later. It's a best practice to tag a subnet if any of the following is true:

• You have multiple clusters that are running in the same VPC.
• You have multiple AWS services that share subnets in a VPC.
• You want more control over where load balancers are provisioned for each cluster.

Resolve multiple subnet discovery errors

1. Confirm that you have at least two subnets in two different Availability Zones. This is a requirement for creating an Application Load Balancer.
   Note: You can create a Network Load Balancer with a single subnet.
2. For each subnet, specify a CIDR block with at least a /27 bitmask (for example: 10.0.0.0/27) and at least eight free IP addresses.
3. Confirm that the tags on the subnets are formatted correctly. For example, the tags can't have any leading or trailing spaces.