How do I resolve the "No space left on device" error from AWS Batch?

Short description

AWS Batch can return a "No space left on device" error for two reasons:

• There's not enough disk space on your Docker volume hosted on Amazon Elastic Block Store (Amazon EBS).
• (For Amazon Linux 1 only) You've reached the default Docker storage limit of 10 GiB.

To resolve the error, first verify that your Docker volume hosted on Amazon EBS has enough disk space. If there's not enough disk space, then increase the size of your Amazon EBS volume. If there is enough disk space, then increase the default Docker storage limit.

Both solutions require creating a new custom launch template and a new compute environment. You can't add a new custom launch template to an existing compute environment in AWS Batch.

Note: The Amazon Elastic Container Service (Amazon ECS) optimized Amazon Linux Amazon Machine Image (AMI) 1 is deprecated. The Amazon ECS optimized Amazon Linux 2 AMI is the default image used when you create a managed compute environment in AWS Batch. For more information, see AMI storage configuration. This article references Amazon Linux 1 for backward compatibility only.

• If you have an AWS Batch ComputeEnvironment that's using ECS Optimized Amazon Linux 1 AMIs to launch Amazon Elastic Compute Cloud (Amazon EC2) instances, it's a best practice to create a new ComputeEnvironment. AWS Batch then uses the latest available ECS Optimized Amazon Linux 2 AMI at the time of creation.
• If you see the "No space left on device" error for jobs running on an Amazon EC2 instance launched using ECS Optimized Amazon Linux 2 AMI, then increase the size of the root volume (/dev/xvda) to make more storage space available for the container.

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version. You can also use AWS CloudShell to run the following commands.

Verify whether your Docker volume hosted on Amazon EBS has enough disk space or not

Follow the instructions in the To determine the available storage for Docker section or AMI storage configuration.

If there isn't enough disk space, then see the To increase the size of Amazon EBS volumes in AWS Batch section of this article.

-or-

If there is enough disk space, then see the (For Amazon Linux 1 only) To increase the default Docker storage limit section of this article.

Increase the size of Amazon EBS volumes in AWS Batch

1.    Create a new custom launch template that resizes (adds "VolumeSize" larger than current) the Docker data volume ( /dev/xvda on default ECS optimized AL2 image) on your AWS Batch compute resources. For instructions, see Launch template support.

Note: If you're using Amazon Linux 1, then make sure that you enter both of the required Docker volumes (/dev/xvda and /dev/xvdcz) for the DeviceName value.

2.    Create a new compute environment that uses the new launch template.

Note: The associated Amazon EC2 instances show the new drive sizes when AWS Batch spins up the instances.

(For Amazon Linux 1 only) To increase the default Docker storage limit

1.    Create a new custom launch template using the MIME multi-part archive format that increases the default Docker storage limit. For more information, see Amazon EC2 user data in launch templates.

Example MIME multi-part file that overrides the default Docker image settings for an AWS Batch compute resource

Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0

--==BOUNDARY==
Content-Type: text/cloud-boothook; charset="us-ascii"
#cloud-boothook
#!/bin/bash
cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=20G"' >> /etc/sysconfig/docker

--==BOUNDARY==--

Important: If you use the AWS CLI or an AWS SDK, you must first base64-encode the user data. Then, submit that string as the value of the UserData parameter when you call the CreateLaunchTemplate API.

2.    Create a new compute environment that uses the new launch template.