Storage benchmarking: Downloading DeepSeek-V3-0324 from Amazon S3 to local storage at 8.1 GiB/s

Customers often need to move large quantities of data from Amazon S3 into local storage, as quickly as possible. As an example, I was recently asked how fast I could download the entire 641.3 GiB DeepSeek-V3-0324 model.

In this article, you will learn how to recreate the test I carried out, and download the entire dataset in around 80 seconds.

Setup

1. Prepare S3 and IAM:
   1. Create an S3 bucket in the AWS region you will test in.
   2. Configure CloudWatch request metrics for your S3 bucket.
   3. Create an IAM instance profile with access to the bucket.
2. Launch and configure your EC2 instance:
   1. I chose the m6in.32xlarge instance, as this has amongst the highest available EBS bandwidth of 100 Gbps, as well as 170 Gbps of network bandwidth from a single interface. I did not use instance storage as I could get higher throughput from EBS with the instance types available to me.
   2. Launch in a public subnet, or if using a private subnet, attach an S3 gateway endpoint to the VPC.
   3. I left the other settings at default, including network configuration, and used the Amazon Linux 2023 AMI.
   4. If using EBS and not instance storage, Create and attach 10 EBS gp3 volumes, each with 100 GiB capacity and configured with 4000 IOPS and 1000 GiB/s of throughput, for a total of almost 10 GiB/s throughput.
      1. Note that 2000 GiB/s gp3 volumes and 4000 GiB/s io2 volumes are also available in some AWS Availability Zones.
   5. Attach the IAM profile you created.
   6. Enable detailed monitoring of the EC2 instance.
3. Stripe your EBS or instance storage volumes into a single volume, and mount it:

# Install mdadm (if not already installed)
sudo dnf install mdadm -y

# Create RAID 0 (striped) array across all 10 NVMe drives
sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=10 /dev/nvme{1..10}n1

# Create filesystem on the RAID array
sudo mkfs.xfs /dev/md0

# Mount the array
sudo mkdir /striped
sudo mount /dev/md0 /striped
sudo chown $USER:$USER /striped 

echo '/dev/md0 /striped xfs defaults,nofail 0 2' | sudo tee -a /etc/fstab

4. Configure the (preinstalled) AWS Command Line Interface (CLI) to optimize throughput:

aws configure set default.s3.multipart_chunksize 16MB
aws configure set default.s3.preferred_transfer_client crt
aws configure set default.s3.target_bandwidth 50GB/s

Note: Setting target_bandwidth to a higher number than the available bandwidth increases the request parallelization driven by the AWS Common Runtime (CRT). This may not be necessary, and can be detrimental if increased too far.

Download the dataset from Hugging Face, and upload to S3

# Make sure the hf CLI is installed
curl -LsSf https://hf.co/cli/install.sh | bash

# Download the model
hf download deepseek-ai/DeepSeek-V3-0324 --cache-dir /striped

# Upload the snapshot to S3, replacing `my-s3-bucket` with your bucket
aws s3 sync /striped/models--deepseek-ai--DeepSeek-V3-0324/snapshots/e9b33add76883f293d6bf61f6bd89b497e80e335/ s3://my-s3-bucket/deepseek-v3-0324/

Perform the download test

# Clear the volume
rm /striped/* -rf

# Download the model to disk
time aws s3 sync s3://my-s3-bucket/deepseek-v3-0324 /striped

Can you beat my 80 second result? Let me know in the comments!