Skip to content

100.0 % of the requests to the ELB are erroring with HTTP 4xx when autoscaling capacity is 0

0

I have an elastic beanstalk environment that I want to shut down nightly at 430p, and start back up at 730a. I have an application load balancer listening to port 443 and 80 that redirects all traffic to port 443 and forwards to a target group. I use a scheduled action in my auto scaling group to set the desired, min, and max capacity to 0.

After the instance is terminated (when the scheduled shut down occurs), I receive the following errors on my elastic beanstalk env that toggle back and forth:

Environment health has transitioned from Unknown to No Data. There are no instances. Auto Scaling group desired capacity is set to zero.

Environment health has transitioned from No Data to Unknown. There are no instances. Auto Scaling group desired capacity is set to zero. 100.0 % of the requests to the ELB are erroring with HTTP 4xx. Insufficient request rate (0.5 requests/min) to determine application health

I thought it was because the listeners were pointing to an empty target group, but when I redirected the rule to be a fixed response I saw the same errors.

Any help or direction appreciated.

2 Answers
1

The issue you're experiencing is related to how Elastic Beanstalk evaluates environment health when your auto-scaling group scales down to zero instances.

When you schedule your environment to shut down by setting the desired, min, and max capacity to 0, the load balancer has no healthy instances to route traffic to. This causes the load balancer to return HTTP 4xx errors for any incoming requests. Elastic Beanstalk's enhanced health monitoring system detects these errors and reports them, causing your environment health to fluctuate between "Unknown" and "No Data" states.

This behavior is expected when there are no instances available to handle requests. The "No Data" state occurs because there are no instances to collect health data from, while the "Unknown" state occurs when Elastic Beanstalk detects that 100% of requests to the load balancer are resulting in 4xx errors.

To address this issue, you can customize the enhanced health rules for your Elastic Beanstalk environment to disregard HTTP 4xx errors originating from the load balancer. As of April 2020, Elastic Beanstalk supports customization of load balancer enhanced health rules, allowing you to ignore these 4xx errors during periods when your environment is intentionally scaled down to zero.

Another approach is to adjust the health check grace period for your Auto Scaling group. This gives instances more time to start up and become healthy before they're subject to health checks, which can help during the morning startup process.

These fluctuating health states don't necessarily indicate a problem with your setup if they only occur during the scheduled downtime when no instances are running. It's the expected behavior when the load balancer has no healthy targets to route traffic to.
Sources
Elastic Beanstalk: ELB Health Check Fails During Auto-Scaling Down | AWS re:Post
Troubleshoot a Classic Load Balancer: Health checks - Elastic Load Balancing
Release: Elastic Beanstalk support for load balancer enhanced health rule customization on April 20, 2020 - AWS Elastic Beanstalk

answered a year ago
EXPERT
reviewed 10 months ago
0

It is possible that you have multiple rules and a different rule is being picked that it sending traffic to the ASG.

Enter image description here

In this example, I only have a single rule that sends traffic to fixed response and it works just fine. So, maybe check all your rules. For example, you might have rules for both port 80 and 443 and only one of those is being updated to provide fixed response.

EXPERT
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.