ECS ELB integration- How does Healthy Threshold Count work if unhealthy tasks are restarted?

Question

Hi and thanks in advance,

I am confused with the Healthy Threshold Count mentioned in the ECS docs (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-healthcheck.html), in relation to integrating load balancers with ECS. That page says "the healthy threshold count only applies when a target is transitioning from an unhealthy state back to a healthy state". The docs also say this does not apply when the task is first starting ("For newly registered targets, only a single successful health check is required to consider the target healthy, regardless of the healthy threshold count setting").

This appears to contradict the following docs pages on ALB/NLB/GLB integration (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/alb.html and .../nlb.html and .../glb.html) which all say "If a service task fails the load balancer health check criteria, the task is stopped and restarted.".

If an unhealthy service task is stopped and restarted, how would the healthy threshold count ever take effect because the task would have been stopped before it has a chance to become healthy again?

Let me know if you understand this better than I; I have submitted feedback to the AWS docs team but no reply.

re:Post Agent · Accepted Answer

When a task in an ECS service fails a load balancer health check, the service scheduler's behavior depends on several factors, including the `maximumPercent` and `desiredCount` parameters.

Here's how the process works:

1. If a task is marked unhealthy due to a failed load balancer health check, the service scheduler will first attempt to start a replacement task.

2. Then one of two things happens:
- If the replacement task becomes healthy (passes its health check), the service scheduler stops the unhealthy task
- If the replacement task also becomes unhealthy, the scheduler will stop either the unhealthy replacement task or the existing unhealthy task to maintain the desired count

The healthy threshold count parameter is relevant in scenarios where a task becomes unhealthy but isn't immediately replaced. For example:

- If the `maximumPercent` parameter limits the scheduler from starting a replacement task first, the unhealthy task might remain running while the scheduler stops tasks one at a time to free up capacity
- In this case, if the unhealthy task resolves its issues, it would need to pass the number of consecutive health checks specified by the healthy threshold count to transition back to healthy status

By default, this threshold is set to 5 consecutive successful health checks, which with the default 30-second interval means it takes 2 minutes and 30 seconds for a task to transition from unhealthy back to healthy. You can optimize this by reducing both the interval (to as low as 5 seconds) and the threshold count (to as low as 2), which would reduce the transition time to just 10 seconds.

For newly registered targets (like when a task first starts), only a single successful health check is required regardless of the healthy threshold count setting.  
**Sources**  
[Update Amazon ECS service parameters - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-parameters.html)  
[Amazon ECS services - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html)  
[Load balancer health check parameters - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/load-balancer-healthcheck.html)  
[Optimize load balancer health check parameters for Amazon ECS - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-healthcheck.html)

ECS ELB integration- How does Healthy Threshold Count work if unhealthy tasks are restarted?

Add your answer

Relevant content