How do I troubleshoot AWS Load Balancer Controller errors in Amazon EKS?

Resolution

Ingress object creation errors

If you try to create an ingress object and you incorrectly configured the webhook service or the service certificate expired, then you might receive one of the following error messages:

"Error from server (InternalError): error when creating "sample.yaml": Internal error occurred: failed calling webhook "mservice.elbv2.k8s.aws": failed to call webhook: Post "https://aws-load-balancer-webhook-service.default.svc:443/mutate-v1-service?timeout=10s": no endpoints available for service "aws-load-balancer-webhook-service""

"Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws ": Post "https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1beta1-ingress?timeout=10s ": x509: certificate has expired or is not yet valid"

To troubleshoot these issues, verify the AWS Load Balancer Controller deployment status, check the service configurations, and validate the certificate expiration dates.

To identify the root cause and resolve the error, complete the following steps:

1. Run the following command to verify that the aws-load-balancer Pods run correctly:

   kubectl get pods -n kube-system | grep aws-load-balancer

   Example output:

   aws-load-balancer-controller-df9b89cf7-hbsrd    1/1       Running    0    26d

2. Run the following command to confirm that the aws-load-balancer-webhook-service exists and that the endpoint exists for aws-load-balancer:

   kubectl get svc -n kube-system

   Example output:

   NAME TYPE                           CLUSTER-IP                EXTERNAL-IP  PORT(S)    AGE
   aws-load-balancer-webhook-service   ClusterIP 10.100.219.187  "none"       443/TCP    86d

   In the output, verify that aws-load-balancer-webhook-service is listed, and check that you assigned a valid CLUSTER-IP address. Verify that the PORT shows 443/TCP for webhook communication.

3. Run the following command to check that you correctly connected the services:

   kubectl get ep -n kube-system

   Example output:

   NAME                                ENDPOINTS          AGE
   aws-load-balancer-webhook-service   172.31.3.28:9443    86d

   Note: In the output, verify that the aws-load-balancer-webhook-service has an ENDPOINTS value. The endpoint port should show 9443 to indicate that the webhook service is correctly configured.

4. Run the following command to check whether the certificate expired:

   kubectl get secret -n kube-system aws-load-balancer-webhook-tls -ojsonpath="{.data.ca\.crt}" | base64 -d | openssl x509 -noout -text

   Example output:

   Validity
           Not Before: [START_DATE]
           Not After : [END_DATE]

   Note: Check that the current date falls between the Not Before and Not After dates. If the Not After date has passed, the certificate has expired.

5. (Optional) If the certificate is expired, then run the following command:

   kubectl apply  --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.13.5/cert-manager.yaml

If you still experience issues, then re-install the AWS Load Balancer Controller to resolve issues for webhook configurations, service endpoints, or certificate management. Uninstall the current AWS Load Balancer Controller installation, and then re-install AWS Load Balancer Controller with Helm.

Note: If you delete the AWS Load Balancer Controller, it doesn't delete the ingress object.

Targets don't register for Network Load Balancer or Application Load Balancer errors

If the targets don't register that a Network Load Balancer or Application Load Balancer exists, then you might receive an error even though you installed the AWS Load Balancer Controller and the service deployed.

Run the following command to check whether the target group binding exists:

kubectl get targetgroupbinding -A

If the target group binding exists, then check the network configuration to make sure that communication is open between the targets and the ELB.

If the target group binding doesn't exist, then create a target group binding.

Example:

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: red-tgb
  namespace: red-ns
spec:
  serviceRef:
    name: red-service
    port: 80
  targetGroupARN: arn:aws:elasticloadbalancing:region:account-id:targetgroup/target-group-name/target-group-id

Note: Replace region with your AWS Region. Replace account-id with your AWS account ID. Replace target-group-name with your target group name. Replace target-group-id with your target group ID.

Connectivity issues with the webhook service

Custom resource definition errors

The AWS Load Balancer Controller must have the custom resource definitions (CRDs) required for the webhook service. For more information, see Custom Resources on the Kubernetes website. If you try to connect without the required CRDs, you might receive an error message that looks similar to the following:

"Error from server (InternalError): error when creating "targetgroupbinding.yaml": Internal error occurred: failed calling webhook "mtargetgroupbinding.elbv2.k8s.aws":
failed to call webhook: Post "https://aws-load-balancer-webhook-service.kube-system.svc:443/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding?timeout=30s":
net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

To resolve this issue, run the following command to confirm that you installed CRDs in the cluster:

kubectl get crds | grep target

Example output:

targetgroupbindings.elbv2.k8s.aws 2023-11-29T14:40:39Z

If the output is empty, then the CRDs are incorrectly installed on the cluster. To install the CRDs, run the following command:

kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/aws-load-balancer-controller/crds/crds.yaml

Validate internal connectivity with the webhook service

It's a best practice to allow the communication between the worker nodes. However, the security groups might have restrictive rules that prevent communication with the webhook service.

In order for the AWS Load Balancer Controller to create and modify resources, the resource must be able to communicate with the service. To allow this communication, confirm that the security group of the worker nodes contain the following rules.

Inbound rules:

Port        Source                            
443         worker-node-security-group-id

Note: Replace worker-node-security-group-id with the ID of your security group.

Outbound rules:

Port        Destination                            
443         worker-node-security-group-id 

Note: Replace worker-node-security-group-id with the ID of your security group.

Validate connectivity from the worker nodes to the ELB service

The VPC endpoint allows communication from the AWS Load Balancer Controller worker nodes to the ELB service endpoints. If your private Amazon EKS cluster doesn't have internet access, then you must set up a Virtual Private Cloud (VPC) endpoint.

The VPC endpoints also create an elastic network interface that you can associate with a security group. Make sure that the security group that's attached to the VPC endpoint contains the following ingress rule:

Port        Source                        
443         worker-node-security-group-id   

Note: Replace worker-node-security-group-id with the ID of your security group.

Invalid CreateTargetGroupInput.Port parameter errors

By default, the controller creates the Application Load Balancer in Instance mode. The Instance mode is compatible with NodePort services but can conflict because Instance mode requires node port allocation. For more information, see Ingress annotations on the Kubernetes website. If the port allocation causes target group creation failures, then you might receive the following error message:

"Failed deploy model due to InvalidParameter: 1 validation error(s) found. - minimum field value of 1, CreateTargetGroupInput.Port."

To resolve, complete the following steps for the appropriate mode for your Application Load Balancer:

For Instance mode, complete the following steps:

1. Set up the type as NodePort:

   apiVersion: v1
   kind: Service
   metadata:
     name: service-name
     namespace: namespace-name
   spec:
     type: NodePort
     ports:
       - protocol: TCP
         port: 80
         targetPort: 80

   Note: Replace service-name with your service name and namespace-name with the name of your namespace.
2. Create the Ingress with Instance mode:

   alb.ingress.kubernetes.io/target-type: instance

For IP mode, complete the following steps:

1. Configure the type as ClusterIP:

   apiVersion: v1
   kind: Service
   metadata:
     name: service-name
     namespace: namespace-name
   spec:
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 80
         targetPort: 80

   Note: Replace service-name with your service name and namespace-name with the name of your namespace.
2. Add the following annotation to your ingress:

   alb.ingress.kubernetes.io/target-type: ip

AWS Load Balancer Controller installation errors

If you install the AWS Load Balancer Controller and the manifest has incorrect permissions, then you might receive an error message similar to the following:

"Warning Failed 8s (x8 over 88s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "aws-load-balancer-controller-5c6c66fc48-49rm4_kube-system(924608a2-da7a-4786-ae74-051bb47ea84e)", container: controller)"

To resolve this error, complete the following steps:

1. Run the following command to edit the AWS Load Balancer Controller deployment:

   kubectl edit deployment -n kube-system aws-load-balancer-controller

2. In the deployment configuration, locate the containers section, and then add the following securityContext settings.
   Example configuration:

   apiVersion: apps/v1 
   kind: Deployment  
   spec:
     template:
       spec:
         containers:
         - name: aws-load-balancer-controller
           image: public.ecr.aws/eks/aws-load-balancer-controller:v2.13.3
           securityContext:  
             runAsNonRoot: true  
             runAsUser: 1000

After you edit the deployment, new Pods launch with the new configuration values.