Resource handler returned message: ECS Deployment Circuit Breaker was triggered. (HandlerErrorCode: GeneralServiceException)

0

Goal: I am attempting to run a simple NodeJS application with two routes: '/' which displays "Hello, World!" and '/health' which returns a status code of 200 along with a message.

Problem: I keep getting this error message: Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'." (RequestToken: 9d7b28d1-2a1f-ed3f-3764-3da55e89b691, HandlerErrorCode: GeneralServiceException).

Tried: I attempted to resolve the issue by removing the existing cluster and service. I then proceeded to create a new cluster and service, starting from scratch, utilizing the provided task definition (shown below).

Certainly: Could someone please assist me in identifying the cause based on the clues I provided here? I need guidance on where to look and what to fix. Your assistance in troubleshooting is greatly appreciated. Thank you for taking the time to help with my issue.

CloudFormation events: Enter image description here

Task definition: nodejs-app-task-revision1.json

{
    "taskDefinitionArn": "arn:aws:ecs:ap-south-1:<ACCOUNT_ID>:task-definition/nodejs-app-task:1",
    "containerDefinitions": [
        {
            "name": "nodejs-app-container",
            "image": "<ACCOUNT_ID>.dkr.ecr.ap-south-1.amazonaws.com/nodejs-server",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "nodejs-app-container-8000-tcp",
                    "containerPort": 8000,
                    "hostPort": 8000,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/nodejs-app-task",
                    "awslogs-region": "ap-south-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            },
            "healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "curl -f http://localhost:8000/health || exit 1"
                ],
                "interval": 30,
                "timeout": 5,
                "retries": 3
            }
        }
    ],
    "family": "nodejs-app-task",
    "executionRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "revision": 1,
    "volumes": [],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.24"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "ecs.capability.container-health-check"
        },
        {
            "name": "ecs.capability.execution-role-ecr-pull"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "3072",
    "runtimePlatform": {
        "cpuArchitecture": "ARM64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2024-02-08T11:24:00.551Z",
    "registeredBy": "arn:aws:iam::<ACCOUNT_ID>:root",
    "tags": []
}
2 Answers
0

Hi, Johnson

I would check to make sure you're not missing any dependencies (for example, environment variables) as it appears the application isn't bootstrapping correctly, as is deemed unhealthy.

Please take a look at this https://repost.aws/knowledge-center/ecs-task-container-health-check-failures

If this helps solve your problem, please accept this as the Accepted Answer, so that others on re:Post may benefit - thank you.

profile pictureAWS
answered 3 months ago
profile picture
EXPERT
reviewed 24 days ago
  • Hi, Eric I have tested running the Dockerized application locally and have successfully passed the health checks. Additionally, I have cross-checked the commands specified in the container definition. However, the issue persists.

0
"healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "curl -f http://localhost:8000/health || exit 1"
                ],

The health check command seems to be using localhost to access the container's health endpoint. However, in ECS, the health check is executed externally, not within the container itself. Therefore, using localhost won't work. You should replace localhost with the IP address or domain name of your container.

athar
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions