S3 support multiple byte ranges download

Question

We are trying to optimize download speed of files residing on S3. Access to those file is granted thru a presigned URL. The application we are building is a desktop application running on windows.

The trick is that we need to download multiple bytes ranges, concurrently.

I then tried to use a multiple range approach using curl. This is where you combine byte ranges for each node (curl_easy_setopt(curl, CURLOPT_RANGE, "10000-20000, 50000-75000")). This is ideal to use because you do not download unnecessary bytes, and you can download large ranges at a time yielding very good download speeds (in theory). However, I was unable to get this to work because the server always downloads the entire file (from what I can tell).

Does S3 support multiple byte ranges?

Answer

S3 does support Byte-Range fetches, it is also recommended like you suggested to get better throughput when fetching different ranges in parallel. This is described in the [Best Practices Design Patterns: Optimizing Amazon S3 Performance Whitepaper](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html).

I also ran a test using an S3 object via a presigned URL and curl through the CLI.

Fetching the entire file:
```bash
curl "URL HERE" -i
HTTP/1.1 200 OK
x-amz-id-2: 000000000
x-amz-request-id: 000000000
Date: Wed, 14 Jun 2023 12:33:38 GMT
Last-Modified: Wed, 14 Jun 2023 12:31:54 GMT
ETag: "000000000"
x-amz-server-side-encryption: AES256
Content-Disposition: inline
Accept-Ranges: bytes
Content-Type: text/plain
Server: AmazonS3
Content-Length: 19

this is a test file
```

Fetching the first two bytes.
```bash
curl "URL HERE" -i -H "Range: bytes=0-1"
HTTP/1.1 206 Partial Content
x-amz-id-2: 000000000
x-amz-request-id: 000000000
Date: Wed, 14 Jun 2023 12:34:10 GMT
Last-Modified: Wed, 14 Jun 2023 12:31:54 GMT
ETag: "000000000"
x-amz-server-side-encryption: AES256
Content-Disposition: inline
Accept-Ranges: bytes
Content-Range: bytes 0-1/19
Content-Type: text/plain
Server: AmazonS3
Content-Length: 2

th
```

I've edited the values returned by `x-amz-id-2`, `x-amz-request-id` and `ETag` to save space as they are not relevant to the snippet.

S3 support multiple byte ranges download

Relevanter Inhalt