During an Elastic Beanstalk rolling update, can an instance be deregistered before confirming that the newly launched version is healthy?
While operating an Elastic Beanstalk environment in production, I accidentally deployed a bundle from another project.
As shown in the screenshot, the deployment was configured to roll out gradually—one instance at a time—across a total of two instances.
Because of differences in project code, the deployment failed on the very first instance, and it took about six minutes for me to notice the mistake and abort the deployment.
During that time, I assumed all requests would be handled normally by the remaining instance, since it was still running.
However, after checking the load balancer metrics, I found that only 50 % of the requests succeeded, while the rest returned service-unavailable errors.
Below are the metrics from the target group associated with the load balancer. I interpret the 50 % success rate as a result of requests being distributed in a round-robin manner when all hosts are marked unhealthy.
- Unhealthy Hosts
- Target 2XXs
- Target connection errors
Because health-check requests from the load balancer are not recorded in the access logs, I am sharing only our internal API logs.
- Why did the instance that was still running correctly also transition to an “unhealthy” state?
- What configuration changes can I make to prevent this issue from happening again?
- Newest
- Most votes
- Most comments
Yes, during an Elastic Beanstalk rolling update, it's possible for an instance to be deregistered before confirming that the newly launched version is healthy, which appears to be what happened in your case.
When Elastic Beanstalk performs rolling updates, it processes instances in batches. For each batch, it terminates the old instances and launches new ones with the updated configuration. There are several factors that could have caused your issue:
-
Health Check Configuration: Elastic Beanstalk uses health checks to determine when an instance is ready to serve traffic. If the health check settings aren't properly configured, or if the HealthCheckSuccessThreshold is set too low, Elastic Beanstalk might consider an instance unhealthy prematurely.
-
Command Timeout: There's a timeout period from when an operation begins to when an instance must transition into a healthy state. The default is 10 minutes. If your application takes longer to start up, the update might fail.
-
Rolling Update Timeout: For each batch, there's a separate timeout (default is 30 minutes). If instances in the batch don't become healthy within this timeout, the operation fails.
To prevent this issue in the future, you can:
-
Adjust your health check settings to ensure they accurately reflect when your application is ready to serve traffic.
-
Increase the command timeout and rolling update timeout if your application needs more time to start up.
-
Consider using immutable updates instead of rolling updates. Immutable updates create a new set of instances with the new version and only switch over when the new instances are fully healthy. This provides a more robust update process, especially for applications that are sensitive to startup time.
-
Ensure you have enhanced health reporting enabled to provide more detailed health information.
-
Configure your rolling updates to maintain a minimum number of instances in service during the update process.
By implementing these changes, you can help ensure that your application remains available during updates, even if a deployment fails.
Sources
[EBS] How to ensure rolling updates actually keep the app up and running constantly? | AWS re:Post
Elastic Beanstalk rolling environment configuration updates - AWS Elastic Beanstalk
Configuration changes - AWS Elastic Beanstalk
Relevant content
- AWS OFFICIALUpdated 10 months ago
Thank you. The answer and the re:Post question included in the sources were helpful.