Skip to content

ECS Task with Fargate and EBS volume fails to deploy

0

I'm trying to deploy an ECS standalone task, without a service with Fargate (not EC2) and an EBS volume for neo4j. The tag being used is neo4j:5.25-community When I try to deploy the task definition, it starts provisioning and then stops within a couple seconds. I have increased the start timeout to 50secs to just in case. No Cloudwatch log stream is being created either.

Is ECS with Fargate and EBS supported? I am using the latest version of Fargate? Is there some way to debug this?

Here's the Task definition json -

{
    "taskDefinitionArn": "arn:aws:ecs:us-west-2:...",
    "containerDefinitions": [
        {
            "name": "neo4j-dev",
            "image": "neo4j:5.25-community",
            "cpu": 1024,
            "memory": 2048,
            "portMappings": [
                {
                    "name": "neo4j-http",
                    "containerPort": 7474,
                    "hostPort": 7474,
                    "protocol": "tcp",
                    "appProtocol": "http"
                },
                {
                    "name": "ne04j-bolt",
                    "containerPort": 7687,
                    "hostPort": 7687,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "NEO4J_apoc_export_file_enabled",
                    "value": "true"
                },
                {
                    "name": "NEO4J_apoc_import_file_use__neo4j__config",
                    "value": "true"
                },
                {
                    "name": "NEO4J_AUTH",
                    "value": "none"
                },
                {
                    "name": "NEO4J_PLUGINS",
                    "value": "'[\"apoc\", \"genai\"]'"
                },
                {
                    "name": "NEO4J_apoc_import_file_enabled",
                    "value": "true"
                },
                {
                    "name": "NEO4J_initial_dbms_default__database",
                    "value": "medicalvault"
                }
            ],
            "environmentFiles": [],
            "mountPoints": [
                {
                    "sourceVolume": "neo4j-data-volume-dev",
                    "containerPath": "/data",
                    "readOnly": false
                }
            ],
            "volumesFrom": [],
            "startTimeout": 50,
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/neo4j-docker-dev",
                    "mode": "non-blocking",
                    "awslogs-create-group": "true",
                    "max-buffer-size": "25m",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            },
            "healthCheck": {
                "command": [
                    "curl -f http://localhost:7474"
                ],
                "interval": 30,
                "timeout": 10,
                "retries": 3,
                "startPeriod": 30
            },
            "systemControls": []
        }
    ],
    "family": "neo4j-docker-dev",
    "executionRoleArn": "arn:aws:iam::127214162859:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "revision": 4,
    "volumes": [
        {
            "name": "neo4j-data-volume-dev",
            "configuredAtLaunch": true
        }
    ],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.28"
        },
        {
            "name": "ecs.capability.container-health-check"
        },
        {
            "name": "ecs.capability.container-ordering"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "2048",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2024-12-05T14:03:34.009Z",
    "registeredBy": "arn:aws:iam::127214162859:root",
    "tags": []
}
3 Answers
0
Accepted Answer

Hello.

It is possible to attach EBS volumes with ECS Fargate.
This is now possible with the January 2024 update.
https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ecs-fargate-integrate-ebs/?nc1=h_ls

Try attaching the IAM policy "AmazonECSInfrastructureRolePolicyForVolumes" to the infrastructure role.
This IAM role is required by Fargate to create an EBS volume.
Infrastructure roles can be specified when configuring the ECS service or when starting a standalone task, not in the task definition.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonECSInfrastructureRolePolicyForVolumes
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/infrastructure_IAM_role.html

EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

EXPERT

reviewed 2 years ago

  • You need to set the appropriate IAM policy for the IAM role you specify when launching a standalone task as shown in the image below. Please check the execution history of the "CreateVolume" API from the CloudTrail event history and see if any errors occur. a

  • This worked, I had created a role with some permissions for the arn:aws:ec2:::volume/* resource but looks like some were missing. The deployment seems to fail with permission errors for some missing permissions and but silently stops for others looks like.

    Thanks!

  • Is it possible to attach an existing EBS volume to a task?

0

The Last Status is not underlined and hence not clickable. It seems that it gets a Stop request immediately after deployment.

Also documentation says EBS can be attached to a standalone task. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html

answered 2 years ago

-1

There are a few issues with your ECS task deployment that are likely causing it to fail:

  1. EBS volumes with Fargate: Unfortunately, AWS Fargate does not support using Amazon EBS volumes directly. Fargate tasks can only use ephemeral storage or Amazon EFS for persistent storage. Your task definition includes an EBS volume, which is not compatible with Fargate.

  2. Volume configuration: The volume you've defined in your task definition (neo4j-data-volume-dev) is not properly configured for Fargate. Fargate tasks require specific volume configurations, such as EFS volumes or bind mounts.

  3. CloudWatch Logs: While you've configured logging, the fact that no log stream is being created suggests that the task is failing before it can even start logging. This is likely due to the incompatible volume configuration.

To resolve these issues and successfully deploy your Neo4j container on ECS with Fargate, consider the following steps:

  1. Remove the EBS volume configuration from your task definition.

  2. If you need persistent storage, consider using Amazon EFS instead. You'll need to create an EFS file system and configure it in your task definition.

  3. If you don't need persistent storage, you can use the ephemeral storage provided by Fargate, but be aware that data will be lost when the task stops.

  4. Ensure that your task execution role has the necessary permissions to create and write to CloudWatch log groups and streams.

  5. Double-check that your VPC and security group configurations allow the necessary network traffic for your Neo4j container.

To debug Fargate task issues:

  1. Check the ECS console for any error messages in the stopped tasks section.

  2. Verify that your container can run locally using Docker to ensure there are no issues with the image itself.

  3. Review the CloudWatch logs once you've resolved the initial deployment issues.

  4. Consider temporarily increasing the logging detail in your Neo4j configuration to get more information during startup.

By addressing these points, particularly the EBS volume incompatibility with Fargate, you should be able to successfully deploy your Neo4j container on ECS with Fargate.

Sources
AWS Fargate for Amazon ECS - Amazon Elastic Container Service
Amazon ECS task definition differences for the Fargate launch type - Amazon Elastic Container Service
How to debug failed ECS deployment? | AWS re:Post

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.