Can't get launched ECS Fargate service to continue to run

0

I'm trying to create an ECS service hosting my .net core 8 service that is accessible from the internet. I have been spending hours on this but I just have no idea how all of the pieces fit together. The plan is to do this: API Gateway -> Load Balancer -> Service Running in ECS.

I've created a task definition and I can successfully create the ECS service with the task and get it running, but it ALWAYS fails ~10 minutes later with a vague error and nothing in any logs giving an indication as to the issue. This is the error: There was an error deploying candorous-backend Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'.". I suspect it has something to do with my port configuration but I don't understand how ports work in ECS Fargate.

My docker image runs beautifully locally in Docker Desktop and I can hit the site no problem locally.

This is my task definition in ECS

{
    "family": "candorous-backend",
    "containerDefinitions": [
        {
            "name": "candorous",
            "image": "**************.dkr.ecr.us-east-1.amazonaws.com/candorous-backend:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "candorous-80-tcp",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                },
                {
                    "name": "candorous-8080-tcp",
                    "containerPort": 8080,
                    "hostPort": 8080,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/candorous-backend",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            }
        }
    ],
    "taskRoleArn": "arn:aws:iam::"**************:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::"**************:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "3072",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}

When creating the service I choose Fargate as the launch type (I don't have any EC2 instances). I then scroll down to Load balancer and select Application Load Balancer. I would rather use a Network Load Balancer but this is the only option that lets me Create a new load balancer as part of the service set up which I need. For the load balancer settings I choose as my container candorous 80:80, I create a new listener for port 80 (HTTP) and a new Target Group of Protocol: HTTP. I only have one VPC so I leave all of the networking options as defaults. These are the only settings I change, then I click Create.

The service is created successfully, then a task is attached and started successfully. The container runs seemingly fine inside the task but then fails a few minutes later.

There is only one log group in my entire AWS account and this is the log output

|   timestamp   |   message
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1704427908641 | dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
| 1704427908641 |       Hosting starting
| 1704427908683 | dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ModelBinderFactory[12]
| 1704427908684 |       Registered model binder providers, in the following order: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DateTimeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.TryParseModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexObjectModelBinderProvider
| 1704427908751 | info: Microsoft.Hosting.Lifetime[14]
| 1704427908751 |       Now listening on: http://localhost:80
| 1704427908751 | dbug: Microsoft.AspNetCore.Hosting.Diagnostics[13]
| 1704427908751 |       Loaded hosting startup assembly ThreadsOfTruth.Service|
| 1704427908751 | info: Microsoft.Hosting.Lifetime[0]
| 1704427908751 |       Application started. Press Ctrl+C to shut down.
| 1704427908751 | info: Microsoft.Hosting.Lifetime[0]
| 1704427908751 |       Hosting environment: Production
| 1704427908751 | info: Microsoft.Hosting.Lifetime[0]
| 1704427908751 |       Content root path: /candorous
| 1704427908751 | dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
| 1704427908751 |       Hosting started
| 1704428514515 | info: Microsoft.Hosting.Lifetime[0]
| 1704428514515 |       Application is shutting down...
| 1704428514516 | dbug: Microsoft.Extensions.Hosting.Internal.Host[3]
| 1704428514516 |       Hosting stopping
| 1704428514527 | dbug: Microsoft.Extensions.Hosting.Internal.Host[4]
| 1704428514527 |       Hosting stopped
David
asked 4 months ago192 views
1 Answer
0

Hi David, from the description of your error it looks like a problem with the health-check of your LoadBalancer. The task is running fine for few minutes until the LoadBalancer health-check failed and the service restart the task until the deploy is failed. Some common errors are:

  • wrong port configuration or wrong health check path
  • wrong security group configuration that do not allow connection from the LB to the targets.

For more information on how to handle this kind of erro please check this article: https://repost.aws/knowledge-center/elb-fix-failing-health-checks-alb

AWS
answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions