HTTP 503 during Rolling Update of managed node group version on EKS with ALB

0

Hello,

I have an EKS cluster with a managed node group, running a web service with an Ingress / ALB (with AWS Load Balancer Controller) in front. On the ALB I set the deregistration_delay.timeout_seconds to 30 seconds and in my K8s web service deployment, I set the terminationGracePeriodSeconds to 70 seconds and I am doing a sleep 40 in the preStop lifecycle hook to achieve zero downtime rollouts/deployments.

This is working great for kubectl rollout restart deployment ... but everytime I update the node group version (Rolling Update in the management console) to the most recent AMI, I get HTTP 503 errors during the rolling update (AWS docs: There are no registered targets in the ALB target group?)

Not sure if it is relevant but the nodes are running Bottlerocket OS 1.15.0 (aws-k8s-1.27) and EKS version is 1.27 (platform version eks.5), the AWS Load Balancer Controller version is v2.6.1 and I also have the elbv2.k8s.aws/pod-readiness-gate-inject=enabled label set in my namespace.

Thank you for any tips on what could be causing this difference between normal deployment rollouts and node group updates and how to achieve zero-downtime for both.

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions