Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

ECS Circuit Breaker is not preventing deployment loop

0

The circuit breaker feature on ECS deployments is not functioning the way I would expect. Here's my setup:

  1. Create an ECS task definition that uses a simple nginx image configured with a health check endpoint that returns a 200 OK response.
  2. Create an ECS cluster and and ECS service with Deployment circuit breaker turned on and use the task definition from step 1.
  3. Configure the ECS service to use an ALB with a configured health check.

At this point the deployment will succeed and there will be a running task. Now we can simulate a deployment that contains some kind of issue by using a new nginx configuration that returns a 500 status from the health check endpoint.

  1. Create a new task definition revision that points to the new image with the bad health check.
  2. Update the ECS service to use the latest task definition revision.

A new deployment will start that will spin up a task for the new task definiton. This task will successfully make it to the Running stage, but the ALB health check will fail. The task will be stopped and a new task will be started to replace it.

At this point I would expect that after a certain number of attempts to start the new task that the circuit breaker would trigger and fail the deployment, however what I'm actually seeing is that the deployment perpetually attempts to create new tasks and never triggers the circuit breaker.

Topics
DevOpsContainers
Tags
Application Load BalancerDevOpsAmazon Elastic Container ServiceContainers
Language
English

asked 2 years ago377 views

1 Answer
1

The behavior you're experiencing with the ECS Deployment Circuit Breaker not triggering as expected is a common issue that can occur due to several factors.

The Circuit Breaker is designed to stop a deployment if it detects repeated failures, but in your scenario, it seems the tasks are reaching the "Running" state before failing the ALB health checks. This can cause the Circuit Breaker to not recognize these as deployment failures.

To address this issue, consider the following:

  1. Ensure your task definition includes a container-level health check that aligns with your ALB health check. This can help ECS detect failures earlier in the deployment process.

  2. Review your Circuit Breaker configuration. The default threshold is typically 3 failures, but you may need to adjust this based on your specific needs.

  3. Check your service's deployment configuration, particularly the minimum healthy percent and maximum percent settings. These can affect how quickly new tasks are considered failed.

  4. Examine your ALB target group settings, especially the health check grace period. This period allows time for your containers to start up before beginning health checks.

  5. Consider implementing a startup probe or initialization process in your container that prevents it from reporting as "healthy" to ECS until it's truly ready to serve traffic.

  6. Review your task's CPU and memory allocations to ensure they're sufficient for your application needs.

If the issue persists after these adjustments, you may need to investigate further by examining detailed logs from your ECS tasks, the ALB, and CloudWatch for any additional insights into why the tasks are failing after reaching the "Running" state.
Sources
ECS Service Deploy Error: ECS Deployment Circuit Breaker was triggered | AWS re:Post
Improving deployment visibility for Amazon ECS services | Containers

answered 2 years ago

EXPERT

reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content