Resource handler returned message: ECS Deployment Circuit Breaker was triggered (HandlerErrorCode: GeneralServiceException)

0

Goal
I'm trying to run a service using a task definition which is a flask server image for backend.

Problem

  • I keep getting this error message - There was an error deploying the backend-flask container. Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'." (RequestToken: ..., HandlerErrorCode: GeneralServiceException)

I tried

  • Removed the cluster and service. Create a cluster, then create a service using the task definition (provided below) from scratch all over again.
  • Attached the AmazonEC2ContainerRegistryReadOnly policy to both Task role and Execution role to make sure that the service can access the ECR for the flask image.
  • Updated the flask Dockerfile (from EXPOSE ${PORT} to EXPOSE 4567)
  • Checked the CloudFormation log but the error message is too general, unable to pinpoint where is wrong (please see the attached image).

Can someone please help me:

  1. how to reason/identify the cause based on the clues I suggested here.
  2. where to look and what to fix.

I appreciate your taking time for my trouble. Thank you.

CloudFormation log Enter image description here

Task definition: backend-flask.json

{
    "family": "backend-flask",
    "executionRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/appServiceExecutionRole",
    "taskRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/appTaskRole",
    "networkMode": "awsvpc",
    "cpu": "256",
    "memory": "512",
    "requiresCompatibilities": [ 
      "FARGATE" 
    ],
    "containerDefinitions": [
      {
        "name": "backend-flask",
        "image": "<ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/backend-flask",
        "essential": true,
        "healthCheck": {
          "command": [
            "CMD-SHELL",
            "python /backend-flask/bin/flask/health-check"
          ],
          "interval": 30,
          "timeout": 5,
          "retries": 3,
          "startPeriod": 60
        },
        "portMappings": [
          {
            "name": "backend-flask",
            "containerPort": 4567,
            "protocol": "tcp", 
            "appProtocol": "http"
          }
        ],
        "logConfiguration": {
          "logDriver": "awslogs",
          "options": {
              "awslogs-group": "cruddur-week6-ecr",
              "awslogs-region": "us-east-1",
              "awslogs-stream-prefix": "backend-flask"
          }
        },
        "environment": [ ...  ],

        ]
      }
    ]
  }
  • I get the same thing - and everything got deleted so there is no way to follow up on the 'health checks'. IN addition, I dont have this problem using http, only when trying to setup https. FARGATE is a great obfuscator.

3 Answers
1
Accepted Answer

Hi Gwen,

is the app healthy? Typically the circuit breaker is triggered when the app does not bootstrap correctly and is deemed as unhealthy.

Start looking into this: https://repost.aws/knowledge-center/ecs-task-container-health-check-failures

Hope it helps

profile picture
EXPERT
answered a year ago
profile picture
EXPERT
reviewed a month ago
  • Hi @alatech, thank you so much for your help. I resolved the issue greatly owing to your lead.

    To benefit other members in the future, here's how I solved it:

      1. Tested the docker file locally.
      1. Turns out, I provided NONE of the env variables in the Dockerfile which are used across my backend app.
      1. I hard-coded the env variables in the Dockerfile.
      1. Uploaded the Dockerfile WITH env vars to ECR.
      1. Tasks are now running on ECS fine and pretty, no problem!

    Thank youuu :D

1

I had the same problem as you. I suggest you to check the service status ,such as "Essential container in task exited" <Exit code: 1>

I built my image on an M1 Mac, then pushed it to ECR (AWS’s version of Docker Hub) for running via ECS, which was using an x86 architecture. To summarise the problem, docker images built for the M1 chip won't work on x86 machines because the way the chips are built is different. Just like a puzzle piece made for a square puzzle won't fit in a round puzzle. The Docker image has been made to fit one type of puzzle (M1 chip), but it won't fit in the other type of puzzle (x86 machine). Link : https://www.padok.fr/en/blog/essential-container-error-ecs

AWS
keithyu
answered 10 months ago
  • This was my case too, after building the image with windows everything works fine.. avoid building image with M1 Chip (apple silicon)

0

Hi, I had a similar issue and found the root cause by looking into the Cloud Trail logs.

I'm posting this as an answer as I believe it may help anyone having issues with the too general 'GeneralServiceException', no matter which resource one is trying to deploy by using CloudFormation.

Go to CloudTrail -> "Event History" and go to "Preferences" -> "Select visible columns" -> enable "Error code". Then filter the list to find the relevant event. If you enter the event and look at the JSON view, you should be able to find a field called "errorMessage", which holds more detailed information as to what have gone wrong.

answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions