Description

CloudFormation stack failed to create AWS::ECS::Service resource with below error:

Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'." (RequestToken: {{ Request_Token}}, HandlerErrorCode: GeneralServiceException)

The deployment circuit breaker determines whether a service deployment will fail if the service can't reach a steady state. If it is turned on, a service deployment will transition to a failed state and stop launching new tasks.

The ECS Deployment Circuit Breaker is a protective mechanism designed to prevent overwhelmed services and resource exhaustion during deployments. This error occurs when the circuit breaker is triggered due to a high rate of failures or issues during the deployment process of an Amazon Elastic Container Service (ECS) service.

Resolution

1. The common root causes for ECS Deployment Circuit Breaker trigger could be:

   1. Insufficient ECS cluster capacity (CPU/memory) to run the desired task count.
   2. Application errors preventing the task from staying in RUNNING state.
   3. Network or security group issues preventing the tasks from being reachable.

2. Based on the root cause identified, take appropriate action such as:

   1. If insufficient cluster capacity, increase the capacity of the ECS cluster.
   2. If application/Image errors, troubleshoot the application/Image code/configuration.
   3. If network/security group issues, review the network and security group settings.
   4. Verify the permissions under the Task Execution IAM Role.

3. Once the root cause is addressed, try updating the CloudFormation stack again to retry the ECS service creation.

4. If the stack is in ROLLBACK_COMPLETE state, please delete the stack and try to create stack again.

Related Information

How the Amazon ECS deployment circuit breaker detects failures - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html

How do I configure and monitor the Amazon ECS deployment circuit breaker? - https://repost.aws/knowledge-center/ecs-configure-monitor-deployment-circuit-breaker

This article was co-authored by Gourav Nauhria and Chethangowda B C. If you have any questions, please feel free to comment on this article or reach out to us.