Can't get launched ECS Fargate service to continue to run

0

I'm trying to create an ECS service hosting my .net core 8 service that is accessible from the internet. I have been spending hours on this but I just have no idea how all of the pieces fit together. The plan is to do this: API Gateway -> Load Balancer -> Service Running in ECS.

I've created a task definition and I can successfully create the ECS service with the task and get it running, but it ALWAYS fails ~10 minutes later with a vague error and nothing in any logs giving an indication as to the issue. This is the error: There was an error deploying candorous-backend Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'.". I suspect it has something to do with my port configuration but I don't understand how ports work in ECS Fargate.

My docker image runs beautifully locally in Docker Desktop and I can hit the site no problem locally.

This is my task definition in ECS

{
    "family": "candorous-backend",
    "containerDefinitions": [
        {
            "name": "candorous",
            "image": "**************.dkr.ecr.us-east-1.amazonaws.com/candorous-backend:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "candorous-80-tcp",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                },
                {
                    "name": "candorous-8080-tcp",
                    "containerPort": 8080,
                    "hostPort": 8080,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/candorous-backend",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            }
        }
    ],
    "taskRoleArn": "arn:aws:iam::"**************:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::"**************:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "3072",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}

When creating the service I choose Fargate as the launch type (I don't have any EC2 instances). I then scroll down to Load balancer and select Application Load Balancer. I would rather use a Network Load Balancer but this is the only option that lets me Create a new load balancer as part of the service set up which I need. For the load balancer settings I choose as my container candorous 80:80, I create a new listener for port 80 (HTTP) and a new Target Group of Protocol: HTTP. I only have one VPC so I leave all of the networking options as defaults. These are the only settings I change, then I click Create.

The service is created successfully, then a task is attached and started successfully. The container runs seemingly fine inside the task but then fails a few minutes later.

There is only one log group in my entire AWS account and this is the log output

|   timestamp   |   message
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1704427908641 | dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
| 1704427908641 |       Hosting starting
| 1704427908683 | dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ModelBinderFactory[12]
| 1704427908684 |       Registered model binder providers, in the following order: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DateTimeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.TryParseModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexObjectModelBinderProvider
| 1704427908751 | info: Microsoft.Hosting.Lifetime[14]
| 1704427908751 |       Now listening on: http://localhost:80
| 1704427908751 | dbug: Microsoft.AspNetCore.Hosting.Diagnostics[13]
| 1704427908751 |       Loaded hosting startup assembly ThreadsOfTruth.Service|
| 1704427908751 | info: Microsoft.Hosting.Lifetime[0]
| 1704427908751 |       Application started. Press Ctrl+C to shut down.
| 1704427908751 | info: Microsoft.Hosting.Lifetime[0]
| 1704427908751 |       Hosting environment: Production
| 1704427908751 | info: Microsoft.Hosting.Lifetime[0]
| 1704427908751 |       Content root path: /candorous
| 1704427908751 | dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
| 1704427908751 |       Hosting started
| 1704428514515 | info: Microsoft.Hosting.Lifetime[0]
| 1704428514515 |       Application is shutting down...
| 1704428514516 | dbug: Microsoft.Extensions.Hosting.Internal.Host[3]
| 1704428514516 |       Hosting stopping
| 1704428514527 | dbug: Microsoft.Extensions.Hosting.Internal.Host[4]
| 1704428514527 |       Hosting stopped
David
gefragt vor 4 Monaten203 Aufrufe
1 Antwort
0

Hi David, from the description of your error it looks like a problem with the health-check of your LoadBalancer. The task is running fine for few minutes until the LoadBalancer health-check failed and the service restart the task until the deploy is failed. Some common errors are:

  • wrong port configuration or wrong health check path
  • wrong security group configuration that do not allow connection from the LB to the targets.

For more information on how to handle this kind of erro please check this article: https://repost.aws/knowledge-center/elb-fix-failing-health-checks-alb

AWS
beantwortet vor 4 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen