How do I configure and troubleshoot the Ingress NGINX Controller in Amazon EKS?

Short description

The Ingress NGINX Controller deploys, configures, and manages pods that contain instances of NGINX, an open-source HTTP and reverse proxy server. These pods are exposed through the controller's service resource. The service resource receives the traffic for the relevant applications that Kubernetes ingress and service resources represent. For more information, see Ingress NGINX Controller on the GitHub website, and NGINX on the NGINX website.

Resolution

Optional: Install Helm. For more information, see Installing Helm on the Helm website. Helm isn't required when you use a YAML manifest to install the Ingress NGINX Controller.

Optional: Install the AWS Load Balancer Controller. This controller is required for you to configure the Network Load Balancer target group target type to the IP target type. The IP target type registers the Ingress NGINX Controller pod IPs to the target group. The instance target type uses the Ingress NGINX Controller service on NodePort to register the instance.

Install the Ingress INGINX controller

Use one of the following methods to install the Ingress NGINX Controller in a Kubernetes cluster:

Use the YAML manifest file that defines all the different components. Use kubectl to create the resources in the manifest.

-or-

Use Helm to deploy the Ingress NGINX Controller through the project's repository chart.

You can expose the Ingress NGINX Controller externally either through an AWS Classic Load Balancer or AWS Network Load Balancer. By default, both methods expose the controller through an internet-facing Classic Load Balancer.

Use Helm to deploy the Ingress NGINX Controller

Use the applicable command to set up and expose the controller:

Internet-facing Classic Load Balancer

helm upgrade --install ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
ingress-nginx

Internal Classic Load Balancer

You must add the following annotation on the Kubernetes Service that exposes the Ingress NGINX Controller:

service.beta.kubernetes.io/aws-load-balancer-internal: true

Example:

Internet-facing Network Load Balancer

To change the Load Balancer from Classic to Network, add the following annotation:

service.beta.kubernetes.io/aws-load-balancer-type: nlb

Example:

helm upgrade --install ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"="nlb" \
ingress-nginx

Internal Network Load Balancer

To provision an internal Network Load Balancer, add the following annotations:

service.beta.kubernetes.io/aws-load-balancer-type: nlb

service.beta.kubernetes.io/aws-load-balancer-internal: true

Example:

helm upgrade --install ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"="nlb" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-internal"="true" \
ingress-nginx

Network Load Balancer in IP mode through the AWS Load Balancer Controller

The cluster must have the AWS Load Balancer Controller installed. For more information, see AWS Load Balancer Controller on the GitHub website.

Use the following annotation to make sure that the AWS Load Balancer Controller takes over provisioning of the Network Load Balancer:

service.beta.kubernetes.io/aws-load-balancer-type: external

By default, the system creates an internal Network Load Balancer. However, you can add the following annotation to create an internet-facing Network Load Balancer with the IP target group target type:

service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

Example:

helm upgrade --install ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"="external" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"="internet-facing" \
ingress-nginx

Use kubectl and YAML manifest files

Download the YAML manifest file. The following command downloads Ingress NGINX Controller version 1.11.2:

curl -Lo ingress-nginx.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.2/deploy/static/provider/cloud/deploy.yaml

Note: You can replace the value with other controller versions. For more information, see Tags on the GitHub website. In some cases, you must annotate the ingress-nginx-controller Kubernetes Load Balancer Service to change the type and properties of the load balancer that the system provisions.

Example:

curl -Lo ingress-nginx.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.2/deploy/static/provider/cloud/deploy.yaml

Internet-facing Classic Load Balancer

You can apply the manifest against the cluster without any modifications. Run the following command:

kubectl apply -f ingress-nginx.yaml

Internal Classic Load Balancer

Complete the following steps:

1. Modify the YAML manifest to add an annotations section that specifies the following annotation:
   service.beta.kubernetes.io/aws-load-balancer-internal: true
   Example:

   ...
   apiVersion: v1
   kind: Service
   metadata:
     ...
     name: ingress-nginx-controller
     namespace: ingress-nginx
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-internal: true
     ...
   type: LoadBalancer
   ...

2. Run the following command to apply the manifest to your cluster:

   kubectl apply -f ingress-nginx.yaml

Internet-facing Network Load Balancer

1. Use the following annotation to create a Network Load Balancer:
   service.beta.kubernetes.io/aws-load-balancer-type: nlb
   Example:

   ...
   apiVersion: v1
   kind: Service
   metadata:
     ...
     name: ingress-nginx-controller
     namespace: ingress-nginx
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-type: nlb
     ...
   type: LoadBalancer
   ...

2. Run the following command to apply the manifest to your cluster:
   service.beta.kubernetes.io/aws-load-balancer-type: nlb
   Example:

   kubectl apply -f ingress-nginx.yaml

Internal Network Load Balancer:

1. Use the following annotations together to create an internal Network Load Balancer:
   service.beta.kubernetes.io/aws-load-balancer-type: nlb
   service.beta.kubernetes.io/aws-load-balancer-internal: true
   Example:

   ...
   apiVersion: v1
   kind: Service
   metadata:
     ...
     name: ingress-nginx-controller
     namespace: ingress-nginx
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-type: nlb
       service.beta.kubernetes.io/aws-load-balancer-internal: true
     ...
   type: LoadBalancer
   ...

2. Run the following command to apply the manifest to your cluster:

   kubectl apply -f ingress-nginx.yaml

Network Load Balancer in IP mode through the AWS Load Balancer Controller

1. Install the AWS Load Balancer Controller in the cluster. For more information, see AWS Load Balancer Controller on the GitHub website. Use the following annotation to make the AWS Load Balancer Controller provision a Network Load Balancer:

   service.beta.kubernetes.io/aws-load-balancer-type: external

2. By default, the system creates an internal Network Load Balancer. To create an internet-facing Network Load Balancer with the target group target type as IP, add the following annotation:
   service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
   Example:

   apiVersion: v1
   kind: Service
   metadata:
     ...
     name: ingress-nginx-controller
     namespace: ingress-nginx
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-type: external
       service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
     ...
   type: LoadBalancer 

3. Run the following command to apply the manifest to your cluster:

   kubectl apply -f ingress-nginx.yaml

Troubleshoot issues

For more information, see Troubleshooting on the GitHub website.

Subnet error

If the Network Load Balancer can't automatically discover the subnet, then you might receive the following error message:

"Reconciler error","controller":"ingress",...,"error":"couldn't auto-discover subnets: unable to resolve at least one subnet"

The AWS Load Balancer Controller uses subnet tags to automatically find subnets to use for load balancers. For AWS Load Balancers, the controller needs at least two subnets in different Availability Zones. For Network Load Balancers, the controller needs at least one subnet. For auto-discovery, you must tag the subnets:

• Tag public subnets with the kubernetes.io/role/elb key. Set the value to 1.
• Tag private subnets with the kubernetes.io/role/internal-elb key. Set the value to 1.
• If you use AWS Load Balancer Controller version 2.1.1 and earlier, then tag public and private subnets with key kubernetes.io/cluster/your-cluster-name. Set the value to owned or shared.

Address field doesn't show kubectl get or describe ingress

When you run the kubectl get ingress your-ingress-name command, the address field might be empty. Or, kubectl describe ingress might not show an assigned address. In either of these scenarios, take the following actions:

• Describe the Ingress resource to verify whether it uses the correct ingressClassName or kubernetes.io/ingress.class annotation. The IngressClass name of the Ingress NGINX Controller must match the value of the ingressClassName field or kubernetes.io/ingress.class annotation. If it doesn't match, then configure the Ingress NGINX Controller's IngressClass resource as the sole default IngressClass for the cluster. For more information, see Default IngressClass on the Kubernetes website.
• Describe the Ingress resource to see if there are errors added to the events by the Ingress NGINX Controller. If there are no events, then the events have reached their Time To Live limit. Or, the Ingress NGINX Controller can't detect the ingresses that it must take action on.
• Run the following command to check the logs of the Ingress NGINX Controller pods for Role-Based Access Control (RBAC) or other related errors:

  kubectl logs ingress-nginx-controller-pod-name -n ingress-nginx-namespace

  Note: Replace ingress-nginx-controller-pod-name with the name of your Ingress NGINX Controller's pod. Replace ingress-nginx-namespace with the name of your Ingress NGINX namespace.

Access and request logs

Review the default log format for the Ingress NGINX Controller to find information about specific requests and responses. To do this, refer to Log format on the GitHub website. In the following example, the default log format matches the following sample log. It shows that the IP address and port of the backend target is 192.168.114.102.80, and the HTTP response from the backend target is 200.

192.168.116.133 - - \[24/Sep/2024:22:14:59 +0000\] "GET / HTTP/1.1" 200 45 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_15\_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 524 0.003 \[default-apache-service-80\] \[\] 192.168.114.102:80 45 0.003 200 ffe584bdeb28959241e8d8408cfc06e5

Related information

Exposing Kubernetes Applications, Part 3: Ingress-Nginx Controller