[EBS] How to ensure rolling updates actually keep the app up and running constantly?

To ensure rolling updates keep your application up and running constantly in Elastic Beanstalk (EBS), there are several factors to consider and steps you can take:

1. Health Check Configuration: Make sure your health check settings are properly configured. Elastic Beanstalk uses health checks to determine when an instance is ready to serve traffic. You might need to adjust the HealthCheckSuccessThreshold option in the aws:elasticbeanstalk:healthreporting:system namespace to change the level at which Elastic Beanstalk considers an instance to be healthy.

2. Command Timeout: Elastic Beanstalk applies a command timeout from when an operation begins to allow an instance to transition into a healthy state. This timeout is set in your environment's update and deployment configuration (in the aws:elasticbeanstalk:command namespace) and defaults to 10 minutes. You may need to increase this timeout if your application takes longer to start up.

3. Rolling Update Timeout: For rolling updates, there's a separate timeout applied to each batch. This is set in the aws:autoscaling:updatepolicy:rollingupdate namespace. If all instances in the batch aren't healthy within this timeout, the operation fails. You might need to increase this timeout.

4. Batch Size and Pause Time: Adjust the batch size and pause time between batches. A smaller batch size and longer pause time can give your application more time to start up and become healthy before the next batch is processed.

5. Enhanced Health Reporting: Ensure you have enhanced health reporting enabled. This provides more detailed health information and can help Elastic Beanstalk make better decisions about instance health.

6. Application Startup Time: If your application takes a long time to start up and begin serving requests, you might need to increase various timeout settings or adjust your application to start faster.

7. Health Check URL: Ensure your health check URL is properly set and that it accurately reflects when your application is ready to serve traffic.

If you've tried these steps and are still experiencing issues, you might want to consider using an immutable update instead of a rolling update. Immutable updates create a new set of instances with the new version, and only switch over when the new instances are fully healthy. This can provide a more robust update process, especially for applications that are sensitive to startup time or have complex initialization processes.

Remember, the goal is to give your application enough time to start up and become healthy before Elastic Beanstalk considers the update complete and removes the old instances. Adjusting these settings should help ensure your application remains available during updates.

Sources
Elastic Beanstalk rolling environment configuration updates - AWS Elastic Beanstalk
Elastic Beanstalk enhanced health reporting and monitoring - AWS Elastic Beanstalk
Immutable environment updates - AWS Elastic Beanstalk

I'm assuming from the question tags you mean Beanstalk, and not just an EBS Volume backed EC2 environment?

Beanstalk is backed by CloudFormation (CFN), and uses this RollingUpdatePolicy

It moves on to the next batch when 2 things happen

1. A cfn-signal is sent by the instance to CloudFormation. This is generally done in the userdata running in the OS, and might be getting sent before the application is actually fully up and running (you can check SignalResource in cloud trail to see the timestamp)
2. Additionally, the instances must be InService in the ASG. You could delay this via a lifecycle hook which you manager, and only complete when the application is fully up and running. However, this would be a lot of ebextension configurations

As the bot answer suggested, an immutable deployment might be the simplest answer since that creates a new ASG and waits for the new instances to be healthy before deleting the old ASG