How do I configure and monitor the Amazon ECS deployment circuit breaker?

6 minute read

I want to automate rollback and receive notifications when my Amazon Elastic Container Service (Amazon ECS) deployment fails.

Short description

To automate rollback and monitor deployments with the Amazon ECS deployment circuit breaker, complete the following steps:

Configure the deployment circuit breaker.
Configure Amazon EventBridge to monitor the deployment circuit breaker.
Test the deployment failure scenario.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Configure the deployment circuit breaker

Complete the following steps:

Create a JSON file with a task definition that's similar to the following example:

{
    "family": "my-task",
    "containerDefinitions": [
        {
            "name": "sample-container",
            "image": "nginx:alpine",
            "essential": true
        }
    ],
    "executionRoleArn": "arn:aws:iam::123456789876:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "256",
    "memory": "512"
}

Note: Replace 123456789876 with your AWS account ID. If you don't have an ecsTaskExecutionRole, then create a task execution role.

To register the task definition, run the following register-task-definition command:

aws ecs register-task-definition \
    --cli-input-json file://taskdef-success.json

Note: Replace taskdef-success.json with your task definition JSON file.

To create an Amazon ECS service with the deployment circuit breaker and rollback activated, run the following create-service command:

aws ecs create-service \
     --cluster default \
     --service-name my-sample-service \
     --deployment-controller type=ECS \
     --desired-count 1 \
     --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}" \
     --task-definition my-task:1 \
     --launch-type FARGATE \
     --network-configuration "awsvpcConfiguration={subnets=[subnet-12345],securityGroups=[sg-12345],assignPublicIp=ENABLED}"

Note: Replace subnet-12345 with your subnet, and sg-12345 with your security groups. You must set deployment-controller to type=ECS because the deployment circuit breaker is available only for rolling update deployments.
If you don't have default cluster, then run the following create-cluster command to create the cluster:

aws ecs create-cluster \
    --cluster-name example-cluster

Note: Replace example-cluster with your cluster name.

Run the following describe-services command to confirm that the Amazon ECS service is in the Steady state:

aws ecs describe-services \
    --cluster default \
    --services my-sample-service | jq '.services[0].events[] | {message}'

You receive an output that's similar to the following example:

{
  "message": "(service my-sample-service) has reached a steady state."
}
{
  "message": "(service my-sample-service) (deployment ecs-svc/1234567890123456789) deployment completed."
}
{
  "message": "(service my-sample-service) has started 1 tasks: (task 2918eb15dd0f4d42affc2a3a07818abf)."
}

Configure EventBridge to monitor the deployment circuit breaker

Complete the following steps:

Run the following create-topic command to create an Amazon Simple Notification Service (Amazon SNS) topic to use as the EventBridge rule target:
```
aws sns create-topic \
    --name my-topic
```
Note: Replace my-topic with your SNS topic name.

To update the topic attributes to allow calls to the required APIs, run the following set-topic-attributes command:

aws sns set-topic-attributes \
    --topic-arn arn:aws:sns:eu-west-1:123456789876:my-topic \
    --attribute-name Policy \
    --attribute-value '{
  "Version": "2008-10-17",
  "Id": "my_topic_policy",
  "Statement": [
    {
      "Sid": "my_topic_default",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:GetTopicAttributes",
        "SNS:SetTopicAttributes",
        "SNS:AddPermission",
        "SNS:RemovePermission",
        "SNS:DeleteTopic",
        "SNS:Subscribe",
        "SNS:ListSubscriptionsByTopic",
        "SNS:Publish"
      ],
      "Resource": "arn:aws:sns:eu-west-1:123456789876:my-topic",
      "Condition": {
        "StringEquals": {
          "AWS:SourceOwner": "123456789876"
        }
      }
    },
    {
      "Sid": "my_topic_for_sns_Publish",
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:eu-west-1:123456789876:my-topic"
    }
  ]
}'

Note: Replace eu-west-1 with your AWS Region, 123456789876 with your account ID, my-topic with your topic name.

To subscribe to the SNS topic with your email, run the following subscribe command:

aws sns subscribe \
    --topic-arn arn:aws:sns:eu-west-1:123456789876:my-topic \
    --protocol email \
    --notification-endpoint example@example.com

Note: Replace eu-west-1 with your Region, 123456789876 with your account ID, my-topic with your topic name, and example@example.com with your email address.

In the subscription confirmation email that you receive, choose Confirm subscription.

To create an EventBridge rule for service deployment failed events, run the following put-rule command:

aws events put-rule \
  --name "EcsServiceDeploymentFailed" \
  --event-pattern "{\"source\":[\"aws.ecs\"],\"detail-type\":[\"ECS Deployment State Change\"],\"detail\":{\"eventName\":[\"SERVICE_DEPLOYMENT_FAILED\"]}}"

To add the SNS topic as the EventBridge rule target, run the following put-targets command:

aws events put-targets \
    --rule EcsServiceDeploymentFailed --targets "Id"="1","Arn"="arn:aws:sns:eu-west-1:123456789876:my-topic"

Note: Replace eu-west-1 with your Region, 123456789876 with your account ID, and my-topic with your topic name.

Test the deployment failure scenario

Complete the following steps:

Create a JSON file an incorrect with image tag in its task definition that's similar to the following one:

{
    "family": "my-task",
    "containerDefinitions": [
        {
            "name": "sample-container",
            "image": "nginx:wrong-image-tag",
            "essential": true
        }
    ],
    "executionRoleArn": "arn:aws:iam::123456789876:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "256",
    "memory": "512"
}

Note: Replace sample-container with your container instance, nginx:wrong-image-tag with an incorrect image tag, and 123456789876 with your account ID. An incorrect image tag causes the deployment to fail.

To register the task definition, run the following register-task-definition command:

aws ecs register-task-definition --cli-input-json file://taskdef-failure.json

Note: Replace taskdef-failure.json with the title of your task definition JSON file.

To update the service with the new task definition and start a new deployment, run the following update-service command:

aws ecs update-service --service my-sample-service --task-definition my-task:2

Note: Replaces my-sample-service with your service and my-task:2 with your task. The new deployment fails because the task can't pull the image. You receive an output that's similar to the following example:

{
    "version": "0",
    "id": "12345abc-2f7c-f86a-e544-a69218eb1446",
    "detail-type": "ECS Deployment State Change",
    "source": "aws.ecs",
    "account": "123456789876",
    "time": "2024-11-19T17:42:41Z",
    "region": "eu-west-1",
    "resources": [
        "arn:aws:ecs:eu-west-1:123456789876:service/default/my-sample-service"
    ],
    "detail": {
        "eventType": "ERROR",
        "eventName": "SERVICE_DEPLOYMENT_FAILED",
        "clusterArn": "arn:aws:ecs:eu-west-1:123456789876:cluster/default",
        "deploymentId": "ecs-svc/9876543210987654321",
        "updatedAt": "2024-11-19T17:42:40.73Z",
        "reason": "ECS deployment circuit breaker: tasks failed to start."
    }
}

To verify that the Amazon ECS service rolled back, run the following describe-services command:

aws ecs describe-services \
    --cluster default \
    --services my-sample-service | jq '.services[0].events[] | {message}'

You receive an output that's similar to the following example:

{
  "message": "(service my-sample-service) has reached a steady state."
}
{
  "message": "(service my-sample-service) (deployment ecs-svc/1234567890123456789) deployment completed."
}
{
  "message": "(service my-sample-service) rolling back to deployment ecs-svc/1234567890123456789."
}
{
  "message": "(service my-sample-service) (deployment ecs-svc/9876543210987654321) deployment failed: tasks failed to start."
}
{
  "message": "(service my-sample-service) has started 1 tasks: (task b808c60616134ec1ac0c656a2bff1ef2)."
}
{
  "message": "(service my-sample-service) has started 1 tasks: (task 846c9aebd9224c2b832a38942cae5ea6)."
}
{
  "message": "(service my-sample-service) has started 1 tasks: (task 7143d03444574f2db2b567d75df3fe72)."
}
{
  "message": "(service my-sample-service) has started 1 tasks: (task 9a6a399770d940a2b442560c02a6a4c0)."
}
{
  "message": "(service my-sample-service) has reached a steady state."
}
{
  "message": "(service my-sample-service) (deployment ecs-svc/1234567890123456789) deployment completed."
}
{
  "message": "(service my-sample-service) has started 1 tasks: (task 2918eb15dd0f4d42affc2a3a07818abf)."
}

Related information

How the Amazon ECS deployment circuit breaker detects failures

Announcing Amazon ECS deployment circuit breaker

Topics: Containers
Tags: Amazon Elastic Container Service
Language: English

AWS OFFICIALUpdated 2 years ago

No comments

Relevant content

Cloudformation update error and rollback error: 'ECS Deployment Circuit Breaker was triggered'
Timotheus
asked 3 years ago
Error occurred during operation 'ECS Deployment Circuit Breaker was triggered
rePost-User-3108449
asked 3 years ago
ECS Circuit Breaker is not preventing deployment loop
Sam
asked 2 years ago
ECS Service Deploy Error: ECS Deployment Circuit Breaker was triggered
rePost-User-8112070
asked 2 years ago
Error deploying service: ECS Deployment Circuit Breaker was triggered
Accepted Answer
mertt
asked 3 years ago
Why did the Amazon ECS deployment circuit breaker set my deployment state to FAILED?
AWS OFFICIALUpdated 7 months ago
Why doesn't my Amazon ECS deployment roll back after it failed?
AWS OFFICIALUpdated 6 months ago
How do I troubleshoot a circuit breaker exception in OpenSearch Service?
AWS OFFICIALUpdated 2 years ago
How do I troubleshoot a circuit breaker exception in OpenSearch Service?
AWS OFFICIALUpdated 5 months ago
ECS Deployment Circuit Breaker was triggered.
EXPERT
Chethangowda B C
published 2 years ago