How can I mount an Amazon EFS volume to AWS Batch in a managed compute environment?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Use a job definition

You can use AWS Batch to specify an Amazon EFS volume in your job definition. For more information, see Introducing support for per-job Amazon EFS volumes in AWS Batch.

Use a launch template

Use a launch template to mount an Amazon EFS volume to an EC2 instance, and then mount the Amazon EFS volume to a container. Or, mount the EFS volume to containers through AWS Batch without the need to create a custom AMI.

Important: When you create an Amazon EFS volume, use the same Amazon Virtual Private Cloud (Amazon VPC) and subnets that are assigned to your compute environment.

Complete the following steps:

1. Create an Amazon EFS file system.

2. Note the file system ID to use later. Example: fs-12345678

3. Create a launch template that includes a user data section and uses the MIME multi-part file format. For more information, see Mime Multi Part Archive on the Cloud-init website.
   Example MIME multi-part file
   The following example MIME multi-part file configures the compute resource to install the amazon-efs-utils package. Then, the file mounts an existing Amazon EFS file system at /mnt/efs:

   MIME-Version: 1.0
   Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
   
   --==MYBOUNDARY==
   Content-Type: text/cloud-config; charset="us-ascii"
   
   packages:
   - amazon-efs-utils
   
   runcmd:
   - file_system_id_01=fs-12345678
   - efs_directory=/mnt/efs
   
   - mkdir -p ${efs_directory}
   - echo "${file_system_id_01}:/ ${efs_directory} efs tls,_netdev" >> /etc/fstab
   - mount -a -t efs defaults
   
   --==MYBOUNDARY==--

   Important: Replace fs-12345678 with your file system ID.

4. Create a launch template file called mount-efs.json. Adjust the size of your volume based on your needs.
   Example:

   {
     "LaunchTemplateName": "user-data",
     "LaunchTemplateData": {
       "BlockDeviceMappings": [
         {
           "Ebs": {
             "DeleteOnTermination": true,
             "VolumeSize": 30,
             "VolumeType": "gp2"
           },
           "DeviceName": "/dev/xvda"
         }
       ],
       "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
     }
   }

   Important: If you add user data to a launch template in the Amazon Elastic Compute Cloud (Amazon EC2) console, then enter the user data as plaintext. Or, upload the user data from a file. If you use the AWS CLI or an AWS SDK, then base64-encode the user data from the Base64 website. Then, submit that string as the value of the UserData parameter when you call CreateLaunchTemplate.

5. To create a launch template based on the mount-efs.json file, run the following create-launch-template command:

   aws ec2 --region us-east-1 create-launch-template --cli-input-json file://mount-efs.json

   Note: Replace us-east-1 with your AWS Region.
   Example:

   {
     "LaunchTemplate": {
       "LaunchTemplateId": "lt-06935eb650e40f886",
       "LaunchTemplateName": "user-data",
       "CreateTime": "2019-12-26T09:40:46.000Z",
       "CreatedBy": "arn:aws:iam::12345678999:user/alice",
       "DefaultVersionNumber": 1,
       "LatestVersionNumber": 1
     }
   }

6. Create a new compute environment and associate the environment with your launch template.
   Note: When AWS Batch spins up instances, the Amazon EFS volume is mounted on the instances.

7. To verify if the Amazon EFS volume is mounted with the container instance, use SSH to connect to the instance that AWS Batch launched. Then, run the following Linux df command:

   df -h

   Example output:

   Filesystem      Size  Used Avail Use% Mounted ondev
   tmpfs        3.9G   92K  3.9G   1% /dev
   tmpfs           3.9G     0  3.9G   0% /dev/shm
   /dev/xvda1       50G  854M   49G   2% /
   127.0.0.1:/     8.0E     0  8.0E   0% /mnt/efs

   Note: /mnt/efs is automatically mounted.

8. Create a job definition in AWS Batch that includes the volume and mount point.
   Example:

   {
     "jobDefinitionName": "userdata",
     "jobDefinitionArn": "arn:aws:batch:us-east-1:12345678999:job-definition/userdata:1",
     "revision": 1,
     "status": "ACTIVE",
     "type": "container",
     "parameters": {},
     "containerProperties": {
       "image": "busybox",
       "vcpus": 1,
       "memory": 1024,
       "command": [],
       "volumes": [
         {
           "host": {
             "sourcePath": "/mnt/efs"
           },
           "name": "efs"
         }
       ],
       "environment": [],
       "mountPoints": [
         {
           "containerPath": "/mnt/efs",
           "sourceVolume": "efs"
         }
       ],
       "ulimits": [],
       "resourceRequirements": []
     }
   }

9. Use the job definition you created to submit an AWS Batch job.

Related information

Reference: Job definition template that uses ContainerProperties