Problems with target group health checks when upscaling to AWS EKS

Question

I'm using an ingresscontroller when configuring AWS EKS.
However, when upscaling, nodes are created without an ingresscontroller. Consequently, kube-proxy lacks iptables, and the previous ingress controller doesn't register endpoints for pods used on the node. This causes health checks on the newly created node's target group to fail.

To resolve this issue, I'm planning to create an ingress controller using a daemonset. I believe this should be done for both alb and nlb to prevent health check failures during upscaling. Is this the correct configuration?

Of course, if the health check pod is lucky enough to be deployed on the upscaling node, the health check will pass. However, there are many cases where this won't happen, so I think configuring a daemonset is the safest approach.

Sunil · Answer

Your idea to create an ingress controller using a DaemonSet is a good approach. This ensures that every node in your cluster, including newly created ones, will have an ingress controller running.

You're correct that this should be done for both ALB and NLB to prevent health check failures during upscaling. Using a DaemonSet will ensure that as new nodes are added to your cluster, they automatically get an ingress controller pod deployed on them.

I am adding some additional information for your reference, just in case.

**Steps to Implement:**
* Create a DaemonSet manifest for your ingress controller.
* Apply the manifest to your cluster using kubectl.
* Ensure that the ingress controller pods are running on all nodes, including newly created ones.

**Additional Considerations:**
* Make sure your ingress controller is compatible with both ALB and NLB.
* Configure appropriate resource requests and limits for the ingress controller pods to ensure they don't consume excessive resources on your nodes.
* Implement proper RBAC (Role-Based Access Control) for the ingress controller to ensure it has the necessary permissions to function correctly.

After implementing this solution, you can verify it by:
* Scaling up your cluster and checking if the new nodes have the ingress controller running.
* Monitoring the health checks of your target groups to ensure they pass for all nodes, including newly created ones.
* Testing your applications to ensure traffic is being properly routed through the ingress.

**Best Practices:**
* Regularly update your ingress controller to the latest stable version.
* Monitor the performance and resource usage of your ingress controller pods.
* Consider using Helm charts for easier deployment and management of your ingress controller if you're not already doing so.

If you continue to experience issues after implementing this solution, I recommend reviewing the AWS EKS documentation for any specific configurations or best practices related to your particular setup. If problems persist, consider reaching out to AWS Support for more tailored assistance.

References:

1. https://docs.aws.amazon.com/eks/latest/userguide/auto-configure-alb.html
2. https://repost.aws/questions/QUg9mGzIMfTsWDk9DCDeo5KA/targets-are-not-created-in-target-groups
3. https://repost.aws/questions/QUOOil0VrWTMCOwd_nrTRXGg/eks-automode-loadbalancer-functionality
4. https://docs.aws.amazon.com/eks/latest/userguide/helm.html

Problems with target group health checks when upscaling to AWS EKS

Add your answer

Relevant content