Can not download files in Fargate task with mounted EFS volume

Question

I am trying to run a container image as a Fargate task that downloads some files from a public S3 folder and saves them to EFS.
I have created the task definition as you can see below, the task role includes both AmazonECSTaskExecutionRolePolicy and AmazonElasticFileSystemClientFullAccess. The security group has full access. The file system is in the same region as the task. I am running the task with user=0 because otherwise, I am getting "Permission denied" errors.

The problem is the files won't download. The download speed is very slow or it times-out. However, if i remove the volume from the task definition everything works, the files are downloaded at 50Mb/s and saved to the local container storage.

```
{
    "taskDefinitionArn": "arn:aws:ecs:eu-central-1:id:task-definition/open-meteo-sync-task-definition:38",
    "containerDefinitions": [
        {
            "name": "open-meteo-sync-container",
            "image": "id.dkr.ecr.eu-central-1.amazonaws.com/open-meteo",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "open-meteo-sync-container-80-tcp",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "command": [
                "sync",
                "ncep_gfs013,dwd_icon_eu",
                "temperature_2m,relative_humidity_2m",
                "--past-days",
                "3"
            ],
            "mountPoints": [
                {
                    "sourceVolume": "open-meteo-data",
                    "containerPath": "/app/data",
                    "readOnly": false
                }
            ],
            "volumesFrom": [],
            "user": "0",
            "readonlyRootFilesystem": false,
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/open-meteo-sync-task-definition",
                    "awslogs-region": "eu-central-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            },
            "systemControls": []
        }
    ],
    "family": "open-meteo-sync-task-definition",
    "taskRoleArn": "arn:aws:iam::id:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::id:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "revision": 38,
    "volumes": [
        {
            "name": "open-meteo-data",
            "efsVolumeConfiguration": {
                "fileSystemId": "fs-id",
                "rootDirectory": "/"
            }
        }
    ],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "ecs.capability.execution-role-ecr-pull"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
        },
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.efsAuth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "ecs.capability.efs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "8192",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2024-04-03T08:40:36.371Z",
    "tags": []
}

```

Answer

It seems that the issue you're facing with slow download speeds or timeouts when trying to download files from a public S3 bucket and save them to an EFS volume mounted to your Fargate task is likely due to the limited resources allocated to the task.

With Fargate, you don't have control over the underlying EC2 instance type that your tasks run on. Instead, Fargate provisions resources based on the CPU and memory values you specify in your task definition. In your case, you've allocated 1 vCPU and 8 GB of memory, which may not be sufficient for your workload, especially if it involves downloading and writing large files concurrently.

Here are a few suggestions that could help improve the performance of your Fargate task:

1. **Increase CPU and Memory Allocation**: Try increasing the [CPU and memory](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-tasks-services.html#fargate-tasks-size) values in your task definition. For example, you could try allocating 4 vCPUs. This should provide more compute power and memory bandwidth, which could potentially improve the download and write speeds.

2. **Use Graviton-based Fargate**: AWS offers Graviton-based Fargate, which uses AWS Graviton2 processors that are designed for high performance and efficiency. If your application is compatible with Arm64 architecture, using Graviton-based Fargate could provide better performance compared to the default x86_64 architecture.

3. **Optimize Network Performance**: Ensure that your task is running in the same VPC and Availability Zone as the EFS file system. This can help reduce network latency and improve performance when writing to the EFS volume.

4. **Optimize EFS Performance**: Consider using an EFS performance mode that suits your workload. For example, if you have a high write workload, you might benefit from using the "Max I/O" performance mode, which is designed for higher levels of throughput.

5. **Optimize S3 Downloads**: If you're downloading large files from S3, you could consider using [S3 Transfer Acceleration](https://aws.amazon.com/s3/transfer-acceleration/) or [S3 Batch Operations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops.html) to improve download speeds.

6. **Use EC2**: If the Fargate performance remains an issue even after trying the above suggestions, you could consider running your task on an EC2 instance instead of Fargate. This would give you more control over the instance type and allow you to select a larger instance with more compute resources.

By optimizing the resource allocation and taking advantage of performance-oriented features like Graviton and EFS performance modes, you should be able to improve the download and write performance of your Fargate task.

Can not download files in Fargate task with mounted EFS volume

Relevant content