Target Group Unhealthy for Windows Application on EKS Despite Healthy Pod and Service

The 504 Gateway Timeout error you're experiencing with your Windows application on EKS is likely related to the Target Group health check failures, despite your pods appearing healthy. This is a common issue with Windows nodes in EKS clusters, and there are several potential causes and solutions to explore.

First, let's understand why your Target Group might be showing as unhealthy despite the pod being healthy:

1. Security Group Configuration: Ensure that the security groups associated with your Windows nodes allow inbound traffic on the NodePort from the ALB's security group. This is a common oversight that prevents the ALB from successfully connecting to your application.

2. Network Configuration: Verify that the network ACLs and routing tables allow proper communication between the ALB and the EKS nodes. Windows nodes sometimes require specific network configurations.

3. Windows-specific Networking Issues: Windows containers on EKS have different networking behaviors compared to Linux containers. The kube-proxy on Windows nodes might not be correctly configuring the iptables rules needed for proper service discovery and load balancing.

4. Ingress Controller Configuration: If you're using the AWS ALB Ingress Controller, ensure it's properly configured for Windows nodes. You might need to use a DaemonSet approach to ensure every node has the necessary components for proper ingress handling.

5. Application Startup Time: Windows applications often take longer to initialize than Linux applications. If your application takes longer to start than the ALB's health check timeout, it could result in failed health checks until the application is fully ready.

To resolve this issue:

1. Verify Target Group Settings: Check your health check path, port, and protocol in the Target Group configuration. Ensure they match your application's actual health endpoint.

2. Adjust Health Check Settings: Consider increasing the health check grace period or timeout settings to accommodate the longer startup time of Windows applications.

3. Implement Readiness Probes: Configure proper readiness probes in your Kubernetes deployment to ensure pods are only considered ready when the application can actually serve traffic.

4. Check Windows Node Configuration: Ensure your Windows nodes have all the necessary components and configurations for proper networking with EKS.

5. Use a DaemonSet for Ingress Controller: Consider deploying your ingress controller as a DaemonSet to ensure every node, including newly created ones, has the necessary components for proper ingress handling.

The 504 Gateway Timeout error you're seeing through CloudFront is likely because the ALB cannot successfully route traffic to your application due to these health check failures. Once you resolve the Target Group health check issues, the 504 errors should be resolved as well.
Sources
EKS Application on Unmanaged Windows Node Group Fails with 504 Error Until Pipeline is Rerun | AWS re:Post
Problems with target group health checks when upscaling to AWS EKS | AWS re:Post
ALB target group healthcheck failure for ECS service on Fargate with multiple ports | AWS re:Post
Why do new EC2 instances initially show as unhealthy in ALB Target Group during both blue-green deployments and autoscaling? | AWS re:Post