S3 support multiple byte ranges download

0

We are trying to optimize download speed of files residing on S3. Access to those file is granted thru a presigned URL. The application we are building is a desktop application running on windows.

The trick is that we need to download multiple bytes ranges, concurrently.

I then tried to use a multiple range approach using curl. This is where you combine byte ranges for each node (curl_easy_setopt(curl, CURLOPT_RANGE, "10000-20000, 50000-75000")). This is ideal to use because you do not download unnecessary bytes, and you can download large ranges at a time yielding very good download speeds (in theory). However, I was unable to get this to work because the server always downloads the entire file (from what I can tell).

Does S3 support multiple byte ranges?

asked a year ago1969 views
1 Answer
1

S3 does support Byte-Range fetches, it is also recommended like you suggested to get better throughput when fetching different ranges in parallel. This is described in the Best Practices Design Patterns: Optimizing Amazon S3 Performance Whitepaper.

I also ran a test using an S3 object via a presigned URL and curl through the CLI.

Fetching the entire file:

curl "URL HERE" -i
HTTP/1.1 200 OK
x-amz-id-2: 000000000
x-amz-request-id: 000000000
Date: Wed, 14 Jun 2023 12:33:38 GMT
Last-Modified: Wed, 14 Jun 2023 12:31:54 GMT
ETag: "000000000"
x-amz-server-side-encryption: AES256
Content-Disposition: inline
Accept-Ranges: bytes
Content-Type: text/plain
Server: AmazonS3
Content-Length: 19

this is a test file

Fetching the first two bytes.

curl "URL HERE" -i -H "Range: bytes=0-1"
HTTP/1.1 206 Partial Content
x-amz-id-2: 000000000
x-amz-request-id: 000000000
Date: Wed, 14 Jun 2023 12:34:10 GMT
Last-Modified: Wed, 14 Jun 2023 12:31:54 GMT
ETag: "000000000"
x-amz-server-side-encryption: AES256
Content-Disposition: inline
Accept-Ranges: bytes
Content-Range: bytes 0-1/19
Content-Type: text/plain
Server: AmazonS3
Content-Length: 2

th

I've edited the values returned by x-amz-id-2, x-amz-request-id and ETag to save space as they are not relevant to the snippet.

AWS
Gary_S
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions