Why can't I mount my Amazon EFS volumes on my Amazon ECS tasks?

Short description

Misconfigurations in the following settings can cause issues when you mount your Amazon EFS volumes on your Amazon ECS tasks:

• The EFS file system
• Security groups and network access control lists (network ACLs)
• Amazon Virtual Private Cloud (Amazon VPC) settings
• Cross VPC settings when your Amazon EFS volume is in a different VPC than your tasks
• AWS Identity and Access Management (IAM) role policies and permissions

Resolution

Determine the cause of the issue

Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook to determine the issue. You must run the runbook in the same AWS Region where your Amazon ECS cluster resources are located. Also, use the most recently failed task ID. If the failed task is part of an Amazon ECS service, then use the most recently failed task in the service. The failed task must be visible in ECS:DescribeTasks during the automation. By default, stopped ECS tasks are visible for 1 hour after they enter the Stopped state.

Based on the output of the automation, use one of the following manual troubleshooting steps.

Check your EFS file system, security group, and network ACL configuration

If you incorrectly configured your EFS file system, security groups, or network ACL, then you receive an error that's similar to the following examples:

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: b'mount.nfs4: Connection timed out' : unsuccessful EFS utils command execution; code: 32"

-or-

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: mount.nfs4: Connection reset by peer : unsuccessful EFS utils command execution; code: 32"

EFS file system

Make sure that you created your EFS file system mount targets and they're available in the same Availability Zone and subnet as your tasks.

To check the mount targets' Availability Zone, complete the following steps:

1. Open the Amazon EFS console.
2. In the navigation pane, choose File systems.
3. Select your file system.
4. Choose Network to display the list of existing mount targets and their network configuration.
5. Note the Availability Zones for the subnet where the Amazon EFS file system mounted volumes.
6. Open the Amazon ECS console.
7. In the navigation pane, choose Clusters.
8. Select your cluster.
9. Choose the Tasks tab.
10. Choose your task ID.
11. Under Configuration, choose the subnet ID.
    Note: This action opens a new tab in your browser on the Amazon VPC console.
12. Check the subnet's Availability Zone to verify that it matches the Amazon EFS file system Availability Zone. If the Availability Zone of the Amazon EFS file system doesn't match, then create a mount target in the correct Availability Zone.

Security groups and network ACLs

The EFS file system mount target security group must allow NFS traffic from your tasks over TCP on port 2049.

To check the port, complete the following steps:

1. Open the Amazon EFS console.
2. In the navigation pane, choose File systems.
3. Select the file system that you want to check.
4. Choose Network to display the list of existing mount targets and their associated security groups. Check the inbound rules to make sure that they allow TCP traffic from task IP addresses on port 2049.
5. (Optional) To modify the associated security groups, choose Manage.

It's a best practice to restrict access on port 2049 based on where your connection comes from. If all clients are within a specific subnet, then use the subnet's IP address range. If all clients are within the VPC that's associated with Amazon EFS, then use the VPC CIDR. If clients are from remote networks, then use an IP address range that matches those networks.

The subnets must allow network traffic between your Amazon EFS file system and your Amazon ECS tasks.

To check the network traffic configuration, complete the following steps:

1. Open the Amazon VPC console.
2. In the navigation pane, choose Your VPCs.
3. Select your VPC.
4. Under Details, for Main network ACL, choose your ID.
5. Make sure that the inbound and outbound rules in your VPC allow NFS traffic over TCP from task IP addresses on port 2049.

Check your Amazon VPC configuration

If you didn't correctly configure your Amazon VPC, then you receive an error that's similar to the following example:

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: Failed to resolve "fs-abcdefgxyz.efs.us-east-1.amazonaws.com" - check that your file system ID is correct, and ensure that the VPC has an EFS mount target for this file system ID."

Activate DNS resolution and hostnames

You must activate DNS resolution and DNS hostnames in the Dynamic Host Configuration Protocol (DHCP) option set of the VPC.

Note: DNS hostnames is deactivated by default.

Make sure that your custom DNS server can resolve your volume's DNS name

It's a best practice to use AmazonProvidedDNS for your VPC DHCP option sets. If you use a custom DNS server instead, then you must configure a conditional DNS forwarder in the DNS server configuration file.

Include the following settings:

• All DNS queries to AWS resources (*.amazonaws.com) must go to the VPC's default DNS server.
• The IP address must be the primary private IPV4 CIDR range that's provisioned to your VPC plus two (x.y.z.2) or 169.254.169.253. For example, for the CIDR 10.0.0.0/16, the VPC's default DNS server IP address is 10.0.0.2.

Configure settings for an Amazon EFS volume that's in a different VPC than your container instance

To mount an Amazon EFS volume from a different VPC than your container instance, you must create a VPC peering connection between the VPCs. Or, create a transit gateway. Then, use the following workaround steps based on where your tasks run.

Amazon EC2 instances

For tasks that run on Amazon Elastic Compute Cloud (Amazon EC2), log in to your EC2 instance, and then install botocore.

Note: efs-utils uses botocore to interact with other AWS services. For installation steps, see Install botocore on the GitHub website.

Only an Amazon provided DNS service in the same VPC as the EFS file system can resolve the Amazon EFS DNS name. To hardcode the Amazon EFS mount's IP address in the machine's /etc/hosts file, run the following command:

echo "mount-target-IP-address fs-id.efs.region.amazonaws.com" | sudo tee -a /etc/hosts

Note: Replace mount-target-IP-address with the mount target IP address, fs-id with the EFS file system ID, and region with your Region. Resources outside the VPC still can't resolve the DNS name.

Fargate tasks

In the following workaround, VPC A is the VPC where your Amazon ECS tasks are running and the Amazon EFS client is located. VPC B is the VPC for your Amazon EFS volume. The workaround uses Amazon Route 53 Resolver not to hardcode data. For information about pricing, see Amazon Route 53 pricing.

Prerequisites:

• VPCs A and B use the Amazon Provided DNS service. Or, your custom DNS server uses a conditional DNS forwarder in the DNS server configuration file.
• The Amazon EFS security group inbound rules allow NFS traffic over TCP on port 2049 for the VPC A CIDR.

If your tasks run on AWS Fargate, then complete the following steps:

1. Create a Route 53 inbound endpoint in VPC B and configure the following settings:
   For VPC in the region-name Region, select VPC B.
   For Security group for this endpoint, select a security group that allows inbound TCP and UDP traffic from VPC A on port 53.
   For IP addresses, create two IP addresses.
   Note: When your server resolves the DNS name, it typically returns the IP address of the mount point that's in the same Availability Zone as your client. When you use this workaround, your DNS queries originate from the inbound endpoint. As a result, the inbound endpoint's Availability Zone determines the returned IP address.
   To return a single IP address for your EFS file system, configure a single mount point. Or, place both IP addresses in the same Availability Zone. If you select different Availability Zones, then the DNS name returns a different IP address based on the inbound endpoint that forwarded your query.
2. Create a Route 53 outbound endpoint in VPC A and configure the following settings:
   For VPC in the region-name Region, select VPC A.
   For Security group for this endpoint, select a security group that allows outbound TCP and UDP traffic to VPC B on port 53.
3. Create a Route 53 resolver rule to forward DNS queries for the Amazon EFS DNS name to the Amazon DNS in VPC B.
   Configure the following settings:
   For Domain name, enter the full DNS name of your EFS file system, for example fs-abcdef.efs.eu-west-1.amazonaws.com.
   For VPCs that use this rule, select VPC A.
   For Outbound endpoint, select the outbound endpoint in VPC A.
   For Target IP addresses, enter the two IP address for the Route 53 inbound endpoint in VPC B.
4. To verify that you can mount your Amazon EFS volume with the DNS name, run the following commands from your client:

   sudo mount.efs fs-abcdefxyz /mnt

   dig fs-abcdefxyz.efs.region.amazonaws.com

   Note: Replace region with your Region, and your fs-abdcexzy with the file system volume ID.

Check your IAM role policies and permissions

If you have issues with your IAM role policies and permissions, then you receive an error that's similar to the following examples:

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: b'mount.nfs4: access denied by server while mounting 127.0.0.1:/' : unsuccessful EFS utils command execution; code: 32"

-or-

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: b'mount.nfs4: mounting fs-xxx.efs.us-east-1.amazonaws.com:/dir failed, reason given by server: No such file or directory' : unsuccessful EFS utils command execution; code: 32"

Permissions that you define in the following resources might control access to the EFS file system:

• Amazon ECS task role IAM policy
• EFS file system policies
• The POSIX permissions that are assigned to your access point
• The network ACL and security groups
  Note: To check your network ACL and security groups, see the Security groups and network ACLs section.

Check your Amazon ECS task permissions

Note: To identify the task execution role ID, check the executionRoleARN value in the task definition JSON file. For more information about how to access the JSON file, see Step 5: Create a task definition.

Make sure that the Amazon ECS task execution role has the required permissions to locate and mount an Amazon EFS file system to your task.

Complete the following steps:

1. Open the IAM console.
2. In the navigation pane, choose Roles.
3. In the search bar, enter the name of your task execution role, and then choose the task execution role.
4. Choose Permissions policies to expand all the attached policies. Make sure that the list includes the following permissions:
   elasticfilesystem:ClientMount
   elasticfilesystem:ClientWrite
   elasticfilesystem:DescribeMountTargets
   elasticfilesystem:DescribeFileSystems

To add the permissions to the task execution role, complete the following steps:

1. Open the IAM console.
2. In the navigation pane, choose Policies.
3. Choose Create policy.
4. For Service, choose EFS.
5. Expand the Actions allowed section.
6. Select the ClientMount, ClientWrite, DescribeMountTargets, and DescribeFileSystems permissions.
7. Select the resources that the role can perform these actions on, and then choose Next.
8. Enter policy name and description, and then choose Create policy.
9. In the navigation pane, choose Roles.
10. In the search bar, enter the name of your task execution role, and then choose the task execution role.
11. Choose Add permissions, and then choose Attach policies.
12. Select your new policy.
13. Choose Add permissions.

Check your EFS file system policy

Complete the following steps:

1. Open the Amazon EFS console.

2. Choose File systems, and then select your file system.

3. Make sure that your File system policy allows the task execution role to mount the EFS volume.
   Example policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "ECS Tasks - EFS Access",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::account-id:role/ecs-task-execution-role-name"
         },
         "Action": [
           "elasticfilesystem:ClientMount",
           "elasticfilesystem:ClientWrite",
           "elasticfilesystem:DescribeMountTargets",
           "elasticfilesystem:DescribeFileSystems",
         ]
       }
     ]
   }

4. To add more IAM permissions to your policy, add a new file system policy. Or, add statements to allow your clients to use your Amazon EFS volume to modify the existing policy.

Check your POSIX permissions

If you use access points with a custom root directory path, then make sure that you have POSIX user and root directory creation permissions. For more troubleshooting information, see Mounting with access point fails.

Related information

Creating EFS file systems

Managing mount targets

How do I mount an Amazon EFS file system on an Amazon ECS container or a task that runs on Fargate?

File system mount fails immediately after file system creation

Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate – part 2