How can I optimize performance when uploading large files to S3 across multiple AWS Regions?

Question

We need the ability to upload large files—between 500 MB and 1 GB per file—to Amazon S3 in under a minute. The files are already compressed, and the files come from our application's end users, who are located across continental United States.

We've already looked at using multiple AWS Regions so that we can send uploads to the same Region that the end user is already in, but that didn't improve performance enough. We also looked at activating S3 Transfer Acceleration, but the S3 Transfer Acceleration was slower than going over the public internet in several locations.

What's the best way to optimize performance when uploading large files to S3 across multiple Regions?

Accepted Answer

To upload a 1 GB file to S3 in less than 1 minute, your application's end users need to achieve an average upload rate of 17 MB per second. Assuming that criteria can be met for your use case, you'll need to plan for a transfer method that can reach an upload rate of 17 MB per second.

When you upload large files to S3, it's a best practice to use [multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html). Using multipart uploads allows you to break down a larger file into smaller parts for quicker upload speeds. For more information, see [How can I optimize performance when I upload large files to Amazon S3?](https://aws.amazon.com/premiumsupport/knowledge-center/s3-upload-large-files/)

**Note:** A multipart upload requires that a single file is uploaded in not more than 10,000 distinct parts. You must be sure that the **chunksize** that you set balances the part file size and the number of parts. For example: If the file size is 1 GB and your average upload rate is 500 KB per second for each TCP connection, then you'd want to configure at least a 34-part upload with a part size that's equal to or smaller than 30 MB.

How can I optimize performance when uploading large files to S3 across multiple AWS Regions?

Relevant content