Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

ECS failed to restarted services despite target groups marked unhealthy

0

I had a case today where ECS did not appear to respond to target groups being marked unhealthy by the ELB.

The event log in the ECS service was simply empty for a period of 6+ hours where EC2 target group healthy monitors were reporting 2 (of 2) unhealthy hosts. The ECS tasks themselves stopped logging messages (for reasons unknown) 6 hours previously.

This looks to me as a break in the path between EC2 target group monitoring and ECS service management automation. Given that I have seen this automation working correctly in the past and nothing relevant has changed in the configuration of the target groups or service, it seems there has to be some other factor at play.

If it happens again, I update this post. I am noting it here just in case others have also noticed a recent degradation in the ability of ECS service management automation to detect and correct task failures.

Topics
Networking & Content DeliveryContainers
Tags
Elastic Load BalancingAmazon Elastic Container Service
Language
English

asked a year ago139 views

1 Answer
0

When a task in an ECS service fails a load balancer health check, the service scheduler should replace it based on specific logic. However, there are several factors that could explain why your unhealthy tasks weren't automatically replaced.

The replacement process depends on the maximumPercent and desiredCount parameters in your service definition. When a task is marked unhealthy, the service scheduler first attempts to start a replacement task before stopping the unhealthy one. If the replacement task becomes healthy, the scheduler stops the unhealthy task. If the replacement task also becomes unhealthy, the scheduler will stop either the unhealthy replacement or the existing unhealthy task to maintain the desired count.

If your maximumPercent parameter is limiting the scheduler from starting replacement tasks first, the scheduler will stop unhealthy tasks one at a time at random to free up capacity before starting replacements. This process continues until all unhealthy tasks are replaced with healthy ones.

For your specific case where no action was taken despite unhealthy target groups, several issues could be at play:

  1. The ECS service might not be receiving the health status information from the load balancer
  2. There might be capacity constraints preventing new tasks from starting
  3. There could be IAM permission issues affecting the task execution
  4. If tasks are running manually (not as part of a service), they would continue regardless of health status

To troubleshoot further, you should check:

answered a year ago

EXPERT

reviewed a year ago

  • Jon
    a year ago

    ECS isn't even reporting the tasks as unhealthy, let alone refusing to start because of lack of capacity.

  • Jon
    a year ago

    This:

    The ECS service might not be receiving the health status information from the load balancer

    appears to be the case, but this appears to be a cause internal to the implementation of AWS and not something that I can control.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content