Compute environment with managed EC2 instance - unable to extend root partition size

Question

I have a need for more storage space in my Compute Environment running a Batch job (Docker image from ECR). I thought it would be simple to just extend the root file system size for the EC2 instance running the job. I found that I should be able to use a Launch Template to assign an EBS volume larger than the default 30G to my Compute Env. This works only partially  - running `lsblk` on the instance (from within the Docker image - as part of the Batch job), I see that the large EBS device is available:

```
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
xvda    202:0    0   8T  0 disk 
`-xvda1 202:1    0   8T  0 part /etc/hosts
```

However `df -h` shows that only the default 30G is partitioned and available for the system:

```
Filesystem      Size  Used Avail Use% Mounted on
overlay          30G  5.1G   25G  17% /
tmpfs            64M     0   64M   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/xvda1       30G  5.1G   25G  17% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs            16G     0   16G   0% /proc/acpi
tmpfs            16G     0   16G   0% /proc/scsi
tmpfs            16G     0   16G   0% /sys/firmware
```

I tried adding a script as part of `UserData` in the Launch template to extend the partition and the file system (following Amazon Q suggestion), but it looks like it's not working - `/dev/xvda1` stays 30G, no matter what I try.

The EC2 instance is a ` m4.2xlarge` I'm using Serverless Framework to define my Batch components, and this is how the Launch Template and Compute Env definitions look like:

```
    ExtendedStorageLaunchTemplate:
      Type: AWS::EC2::LaunchTemplate
      Properties:
        LaunchTemplateData:
          BlockDeviceMappings:
          - DeviceName: /dev/xvda
            Ebs:
              VolumeSize: 8192
              VolumeType: gp2
              DeleteOnTermination: true
          UserData:
            Fn::Base64: !Sub |
              MIME-Version: 1.0
              Content-Type: multipart/mixed; boundary="==BOUNDARY=="

--==BOUNDARY==
              Content-Type: text/cloud-config; charset="us-ascii"

#cloud-config
              cloud_final_modules:
              - [scripts-user, always]

--==BOUNDARY==
              Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
              set -e
              # Extend the partition
              growpart /dev/xvda 1
              # Extend the file system (assuming it's ext4, adjust if using xfs)
              resize2fs /dev/xvda1

--==BOUNDARY==--
    ComputeEnvironmentExtendedStorage:
      Type: AWS::Batch::ComputeEnvironment
      Properties:
        ComputeEnvironmentName: process-layer-service-compute-environment-extended-storage
        Type: MANAGED
        ServiceRole: !Ref BatchServiceRole
        ComputeResources:
          InstanceRole: !GetAtt ecsInstanceProfile.Arn
          InstanceTypes:
            - m4.2xlarge
          Type: EC2
          MaxvCpus: 1024
          MinvCpus: 0
          LaunchTemplate:
            LaunchTemplateId: !Ref ExtendedStorageLaunchTemplate
            Version: !GetAtt ExtendedStorageLaunchTemplate.LatestVersionNumber
          SecurityGroupIds:
            - sg-XXXXXXX
          Subnets:
            - subnet-XXXXXXX
        State: ENABLED
```

How can I make this work? Maybe this is a wrong approach?

Gary Mclean · Answer

AWS Batch supports Launch Templates where you can customise the EC2 when launched. Here you can define the EBS volume size

https://docs.aws.amazon.com/batch/latest/userguide/launch-templates.html

Praveen Muppala · Answer

Hello Pawel,

You might want to consider using a secondary volume instead of expanding the root volume. You can refer to this [GitHub resource](https://github.com/vfrank66/awsbatchlaunchtemplate/blob/master/aws-batch-launch-ami.yaml).

Alternatively, you can create a custom AMI with an 8TiB root volume and include that custom AMI in your launch template definition.

Pawel · Answer

I'll share here the way I solved my issue with help from an Amazon solutions architect - it might be useful for someone in the future.

As suggested by Praveen in one of the answers here, I went for using a secondary volume on my EC2 Compute Environment, defined by a Launch Template. I later mount it as a volume in the Docker image used for my Batch Job Definition. The key point is that it's not enough to define an EBS volume and attach it to a device in the Launch Template. The volume also needs to be partitioned and and mounted in the file system to be available for use. Partitions larger than 2TiB [can only be obtained with GPT partitioning scheme](https://docs.aws.amazon.com/ebs/latest/userguide/volume_constraints.html) - for this some extra tools not available at launch on the EC2 instance need to be added (`xfstools`).

This can be done using a User Data script defined in the Launch Template, taking care to include MIME headers correctly. Something like this does the job:

```
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash

# Update package list and install xfsprogs
yum update
yum install -y xfsprogs

# Create XFS filesystem on /dev/xvdb
mkfs.xfs /dev/xvdb

# Create mount point directory
mkdir -p /data

# Mount the volume
mount /dev/xvdb /data

# Add entry to fstab for persistent mounting
echo "UUID=$(blkid -s UUID -o value /dev/xvdb) /data xfs defaults,nofail 0 2" >> /etc/fstab

echo "EBS Volume mounted successfully"

--==MYBOUNDARY==--
```

Didier_Durand · Answer

Hi,

This KC article is probably the answer to your question: https://repost.aws/knowledge-center/extend-linux-file-system

Follow this guidance to be able to extend your partition with available disk space.

Best,

Didier

Compute environment with managed EC2 instance - unable to extend root partition size

Relevant content