How to consume EphemeralStorage in an AWS Batch job's container?

Question

Hi experts,

I am running my docker container in Fargate via AWS Batch. I read that one can specify the EphemeralStorage option in the job definition. Question is how to consume this ephemeral storage in the container being executed? Do I need to define a volume to access it? Any help would be appreciated.

Thanks.

Didier Durand · Answer

Hi,

Baiscally, ephemeral storage is disk space that you get only for the duration of you AWS batch job. It gets scratched after your job completes. You can get up to 200 GB of such storage with this featue.

This KB article explains how to configure and make use of this ephemeral storage in your AWS Batch setup: https://repost.aws/knowledge-center/ecs-fargate-increase-disk-space,

It may also help to read the announcement: https://aws.amazon.com/about-aws/whats-new/2023/03/aws-batch-configurable-ephemeral-storage-fargate/

Best,

Didier

Leonardo Rangel · Answer

Hi,
The ephemeralStorage is mounted at the root (`/`) level. 
This means that you can consume it anywhere you want inside the container running. It does not have an specific path to use it.

You can verify this by running the following command in the console:
```bash
df -h
```

This will output something as the following:
```bash
sh-4.2# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         158G  111G   41G  74% /
tmpfs            64M     0   64M   0% /dev
shm             1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/nvme1n1    158G  111G   41G  74% /etc/hosts
/dev/nvme0n1p1  4.9G  2.1G  2.8G  43% /managed-agents/execute-command
tmpfs           1.9G     0  1.9G   0% /proc/acpi
tmpfs           1.9G     0  1.9G   0% /sys/firmware
```

You can set a minimum of `21GB` to a maximum of `200GB`.
https://docs.aws.amazon.com/batch/latest/APIReference/API_EphemeralStorage.html

kranthi putti · Answer

Hello,

To consume ephemeral storage in an AWS Batch job's container, you can follow these steps:

**Specify Ephemeral Storage in Job Definition:**

* Add the ephemeralStorage parameter in your job definition.

For example:

```
"containerProperties": {
    "image": "your-docker-image",
    "resourceRequirements": [
        {
            "type": "VCPU",
            "value": "2"
        },
        {
            "type": "MEMORY",
            "value": "4096"
        }
    ],
    "ephemeralStorage": {
        "sizeInGiB": 100
    }
}

```
**Access Ephemeral Storage in the Container:**

* Ephemeral storage is automatically mounted to the container at /scratch.
* You can directly read from and write to /scratch within your container.

```
docker run your-docker-image sh -c "echo 'Hello World' > /scratch/hello.txt"

```
This will utilize the specified ephemeral storage location in your AWS Batch job's container.

How to consume EphemeralStorage in an AWS Batch job's container?

Add your answer

Relevant content