Why can't I mount my Amazon EFS volumes on my AWS Fargate tasks?

Short description

Amazon EFS provides a persistent storage solution for your Fargate tasks to share files and data across different tasks.

The following issues might prevent you from mounting your Amazon EFS volumes on your Fargate tasks:

• The Amazon EFS file system isn't configured correctly.
• The AWS Identity and Access Management (IAM) role for the Amazon Elastic Container Service (Amazon ECS) task doesn't have the required permissions.
• There are issues related to network and Amazon Virtual Private Cloud (Amazon VPC) configurations.

To troubleshoot the errors for Amazon Elastic Container Service (Amazon ECS) tasks that fail to start, use the AWSSupport-TroubleshootECSTaskFailedToStart runbook. Then, refer to the relevant troubleshooting steps for your issue.

Resolution

Find the task that failed to start

Important:

• Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook in the same AWS Region where your ECS cluster resources are located.
• When using the runbook, you must use the most recently failed Task ID. If the failed task is part of an Amazon ECS service, then use the most recently failed task in the service. The failed task must be visible in ECS:DescribeTasks during the automation execution. By default, stopped ECS tasks are visible for 1 hour after entering the Stopped state. Using the most recently failed task ID prevents the task state cleanup from interrupting the analysis during the automation.

For instructions on how to initiate the runbook, see AWSSupport-TroubleshootECSTaskFailedToStart. Based on the output of the automation, use one of the following manual troubleshooting steps.

Troubleshoot the task based on your error message

When you try to mount your EFS volume on your Fargate task, you might get one of the following errors:

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: b'mount.nfs4: Connection timed out' : unsuccessful EFS utils command execution; code: 32"

You get these errors when your Fargate task can't connect to the EFS filesystem because of connection timing. To resolve this error, complete the following troubleshooting steps:

1. Open the Amazon EFS console.
2. In the navigation pane, choose File systems.
3. Choose the file system that you want to check by choosing its Name or the File system ID.
4. Choose Network to display the list of existing mount targets.
5. Choose Manage.

You can view the security group and the security group's inbound rules for the mount targets.

Be sure that the inbound rule for the security group allows traffic from the Fargate task security group on port 2049. Confirm that network traffic is allowed at the subnet level. To do this, verify that the network access control list allows traffic between the file system and task. If the traffic isn't allowed, then modify the rules accordingly. For more information, see Security in Amazon Virtual Private Cloud.

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: mount.nfs4: Connection reset by peer : unsuccessful EFS utils command execution; code: 32"

You get this error due to one of the following reasons:

• You mounted the EFS file system immediately after creating the file system.
• The security group for the mount target doesn't allow inbound traffic from Fargate tasks on port 2049.
• You're using AWS App Mesh, and outbound to port 2049 is blocked because of proxy rules.

To troubleshoot this error, follow these steps:

• Up to 90 seconds can elapse for the DNS records to propagate completely in an AWS Region after creating a mount target. If you're programmatically creating and mounting the file systems, such as with an AWS CloudFormation template, then implement a wait condition.
• Confirm that the inbound security group rule that's attached to the EFS file system mount targets allows traffic on port 2049 from Fargate tasks.
• If you're using App Mesh, then make sure that your proxy configuration that's specified in the TaskDefinition includes 2049 as EgressIgnoredPorts.

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: Failed to resolve "fs-xxxxxxxxxxx.efs.us-east-1.amazonaws.com" - check that your file system ID is correct"

You get this error due to one of the following reasons:

• The EFS file system mount target isn't created or available in an Availability Zone where Fargate tasks are launched.
• You're using a custom DNS server for the VPC.
• The VPC DNS hostnames are turned off. DNS hostnames are turned off by default.

To resolve this error, try the following steps:

• Be sure that the EFS file system mount target is in the same Availability Zone as the Fargate task. You can view the Availability Zone, subnet, and security group of the mount target in the Amazon EFS console. Then, verify that the mount target uses the same Availability Zone and subnet as the Fargate task.
• If you specified a custom DNS server for your VPC DHCP options instead of AmazonProvidedDNS, then be sure to configure conditional DNS forwarders. The DNS forwarders must send the DNS queries of AWS resources (*.amazonaws.com) to the VPC's default DNS server at VPC CIDR .2 or 169.254.169.253. For more information, see How to set up DNS resolution between on-premises networks and AWS using AWS Directory Service and Microsoft Active Directory.

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: b'mount.nfs4: access denied by server while mounting 127.0.0.1:/' : unsuccessful EFS utils command execution; code: 32"

You get this error when the following policies and permissions deny access to the file system:

• The file system policy
• The task role policy
• The POSIX file system level permissions

Access to an EFS file system might be controlled by permissions that are defined in the following resources:

• The network access control list
• Security groups
• EFS file system policies
• ECS task role IAM policy
• A POSIX file

For more information, see Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate – Part 2.

To troubleshoot this error, check if the file system policy or the ECS task role IAM policy denies access to the file system. If these policies deny permissions, then modify the policies to grant permissions to access the file system. If the file system policy doesn't exist, then access to the file system is granted by default to all principals during creation.

Related information

Creating Amazon EFS file systems

Creating and managing mount targets and security groups

How do I mount an Amazon EFS file system on an Amazon ECS container or task running on Fargate?

File system mount fails immediately after file system creation