How do I mount an existing Amazon FSx for Lustre file system to AWS Batch in a managed compute environment?

Short description

Use an Amazon Elastic Compute Cloud (Amazon EC2) launch template with AWS Batch. This configuration allows you to mount an existing Amazon FSx for Lustre file system to your containers without creating custom AMIs.

Important: When you create the file system, use the same Amazon Virtual Private Cloud (Amazon VPC) and subnets that are assigned to your compute environment.

Resolution

1.    Create an Amazon FSx for Lustre file system. Then, choose either a persistent or scratch file system deployment type.

Tip: Use persistent file systems for longer-term storage and workloads. Use scratch file systems for temporary storage and shorter-term data processing.

2.    Copy your file system ID (for example, fs-12345678). You need the file system ID to run your launch template.

3.    Create a launch template that includes a user data section and uses the MIME multi-part file format. For more information, see Mime Multi Part Archive on the Cloud-init website.

Example Amazon Linux 2 MIME multi-part file

Important: Replace fs-12345678 with your file system ID. Replace us-east-1 with your AWS Region. Replace xxxxxxxx with your eight-character Amazon FSx mount name.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"

runcmd:
- file_system_id_01=fs-09c9ae4593b53a741
- region=us-east-1
- fsx_directory=/scratch
- fsx_mount_name=xxxxxxxx
- amazon-linux-extras install -y lustre2.10
- mkdir -p ${fsx_directory}
- mount -t lustre -o noatime,flock ${file_system_id_01}.fsx.${region}.amazonaws.com@tcp:/${fsx_mount_name} ${fsx_directory}

--==MYBOUNDARY==--

Note: This example MIME multi-part file configures the compute resource to install the Lustre 2.10 package from the Extras library. The file also mounts an existing Amazon FSx for Lustre file system at /scratch. For installation instructions for other Linux distributions, see Installing the Lustre client.

4.    Launch the template to invoke the user data.

5.    Create a file called mount-fsx-lustre.json.

Note: Adjust the volume size based on your use case.

Example Amazon Linux 2 launch template

{
  "LaunchTemplateName": "user-data",
  "LaunchTemplateData": {
    "BlockDeviceMappings": [
      {
        "Ebs": {
          "DeleteOnTermination": true,
          "VolumeSize": 30,
          "VolumeType": "gp2"
        },
        "DeviceName": "/dev/xvda"
      }
    ],
    "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
  }
}

Example Amazon Linux 1 launch template

{
  "LaunchTemplateName": "userdata",
  "LaunchTemplateData": {
    "BlockDeviceMappings": [
      {
        "Ebs": {
          "DeleteOnTermination": true,
          "VolumeSize": 8,
          "VolumeType": "gp2"
        },
        "DeviceName": "/dev/xvda"
      },
      {
        "Ebs": {
          "DeleteOnTermination": true,
          "VolumeSize": 22,
          "VolumeType": "gp2"
        },
        "DeviceName": "/dev/xvdcz"
      }
    ],
    "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
  }
}

Note: If you add user data to a launch template in the Amazon EC2 console, then make sure that you do one of the following: Paste in the user data as plaintext. -or- Upload the user data from a file.

If you use the AWS Command Line Interface (AWS CLI) or an AWS SDK, you must first base64-encode the user data. Then, submit that string as the value of the UserData parameter when you call CreateLaunchTemplate.

6.    Run the following AWS CLI command to create a launch template based on the mount-fsx-lustre.json file that you created in step 5:

Important: Replace us-east-1 with your AWS Region.

aws ec2 --region us-east-1 create-launch-template --cli-input-json file://mount-fsx-lustre.json

Example create-launch-template command output

{
  "LaunchTemplate": {
    "LaunchTemplateId": "lt-08cb09d54bcf551f3",
    "LaunchTemplateName": "fsx-test",
    "CreateTime": "2020-06-30T17:13:22.000Z",
    "CreatedBy": "arn:aws:iam::12345678999:user/test",
    "DefaultVersionNumber": 1,
    "LatestVersionNumber": 1
  }
}

7.    Create a new compute environment and associate that environment with your launch template.

Important: By default, AWS Batch managed compute environments use an approved version of the Amazon Elastic Container Service (Amazon ECS)-optimized AMI for compute resources. You must explicitly set the AMI ID for Amazon ECS-optimized Amazon Linux 2 AMIs and other Linux distributions.

When AWS Batch spins up instances, the Amazon FSx for Lustre file system is now mounted on the container instances.

8.    Check if the file system is mounted with the container instance by using SSH to connect to the instance launched by AWS Batch. Then, run the following Linux df command:

$ df -h

Example df command output

Filesystem                                  Size      Used    Avail    Used%   Mounted on
devtmpfs                                    3.9G        0       3.9G     0%       /dev
tmpfs                                       3.9G        0       3.9G     0%       /dev/shm
tmpfs                                       3.9G     448K       3.9G     1%       /run
tmpfs                                       3.9G        0       3.9G     0%       /sys/fs/cgroup
/dev/xvda1                                   30G     4.2G        25G    15%       /
172.31.79.79@tcp:/xxxxxxxx                  1.1T     4.5M       1.1T     1%       /scratch
tmpfs                                       798M       0        798M     0%       /run/user/1000

Note: /scratch is mounted automatically.

9.    Create a job definition in AWS Batch that includes the volume and mount point.

Example AWS Batch Job definition

{
  "jobDefinitionName": "Fsx-sample",
  "jobDefinitionArn": "arn:aws:batch:us-east-1:12345678999:job-definition/userdata:1",
  "revision": 1,
  "status": "ACTIVE",
  "type": "container",
  "parameters": {},
  "containerProperties": {
    "image": "busybox",
    "vcpus": 1,
    "memory": 1024,
    "command": [],
    "volumes": [
      {
        "host": {
          "sourcePath": "/scratch"
        },
        "name": "Scratch"
      }
    ],
    "environment": [],
    "mountPoints": [
      {
        "containerPath": "/scratch",
        "sourceVolume": "Scratch"
      }
    ],
    "ulimits": [],
    "resourceRequirements": []
  }
}

10.    Submit an AWS Batch job using the job definition that you created in step 9.