Streaming data of unknown size to Amazon S3 with multipart upload
Learn how to upload a stream of unknown size to Amazon Simple Storage Service (Amazon S3) with the multipart upload API, and size each part at run time to fit a memory budget.
Introduction
An energy customer streams seismic data to Amazon S3. Each stream's size isn't known until the stream ends, and some streams reach multiple terabytes. A single PutObject API request can't upload this data because it requires a Content-Length header up front.
This article shows how to upload a stream of unknown size with the multipart upload API and the AWS SDK for Python (Boto3). The writer sizes each part at run time, so memory use stays within a fixed budget.
Solution overview
This solution came out of an AWS Enterprise Support engagement. The following sections describe the customer's requirements and how multipart upload meets them.
Customer scenario
The customer's pipeline runs in a resource-constrained environment that dynamically reallocates memory. Streams range from a few hundred KB to multiple terabytes, with an average of about 5 GB. The writer can't stage data on local disk, and it must size each part from the memory that's available at run time.
The customer's AWS Technical Account Manager (TAM) compared this requirement with the managed clients and recommended that the customer call the multipart upload API directly. The next section explains when direct control of the upload is worth the extra code.
When to control the upload yourself
Some AWS SDKs run the multipart upload for you. These include the S3 Transfer Manager and the AWS Common Runtime (CRT)-based S3 client in the AWS SDK for Java 2.x, and the upload_fileobj method in Boto3. For examples, see Upload a stream of unknown size to an Amazon S3 object using an AWS SDK.
The managed clients choose one part size when the transfer starts and use it for the whole transfer. When you don't know the size, that part size must cover the largest stream that you might receive. For example, a 10 TB stream requires parts of at least 1 GB to stay under the 10,000-part limit, so every stream uses 1 GB parts. The clients also buffer several parts to upload them in parallel. For a stream that it can't read again, the Boto3 transfer manager buffers at least 10 parts in memory by default. Each part is 1 GB in size, for a total of at least 10 GB.
Consider calling the multipart upload API directly in the following situations:
- Stream sizes vary widely, and a part size that you choose for the largest stream wastes memory on the rest.
- You have limited or shared memory, so you must size each part at run time.
- You require a strict memory ceiling, and you accept lower throughput in return.
How multipart upload handles a stream of unknown size
Multipart uploads don't require the object's total size. Each part declares its own Content-Length only. The writer receives the Content-Length for each part from the bytes in its buffer. The optional x-amz-mp-object-size header on the CompleteMultipartUpload API operation declares the total size.
The upload uses the following API calls:
- The CreateMultipartUpload API operation starts the upload and returns an upload ID, UploadId. The request has no size parameter.
- The UploadPart API operation sends one part and returns an entity tag, ETag. Its Content-Length header covers that part only.
- CompleteMultipartUpload sends the ordered part numbers and ETag values. Amazon S3 joins the parts and calculates the object's size.
Each part must meet the following requirements:
- Every part except the last must be between 5 MiB and 5 GiB. The last part can be smaller than 5 MiB.
- An upload can have at most 10,000 parts.
In the example, a Unix pipe connects two programs. A producer writes data to standard output (stdout). A writer reads it from standard input (stdin), buffers it, and uploads each full buffer as a part.
Solution implementation
Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.
Prerequisites
To follow along, you must have the following tools:
- Python 3.10 or later, and Boto3
- The AWS Command Line Interface (AWS CLI)
- AWS credentials that can upload, download, and delete objects, and cancel multipart uploads, in an S3 bucket that you own
- An S3 Lifecycle rule on that bucket that deletes incomplete multipart uploads after 1 day. For instructions, see Configuring a bucket lifecycle configuration to delete incomplete multipart uploads.
Step 1: Create the producer
The producer program, streamer.py, stands in for any source that keeps producing data. It writes a batch of 20,000 JSON log events, about 2 MB, to stdout every second until you stop it.
Save the following code as streamer.py:
import json
import random
import sys
import time
from datetime import datetime, timezone
PATHS = ["/api/orders", "/api/users", "/api/cart", "/health"]
try:
while True:
lines = []
for _ in range(20000):
event = {
"ts": datetime.now(timezone.utc).isoformat(),
"method": random.choice(["GET", "POST"]),
"path": random.choice(PATHS),
"status": random.choice([200, 200, 200, 404, 500]),
"latency_ms": random.randint(5, 250),
}
lines.append(json.dumps(event))
sys.stdout.write("\n".join(lines) + "\n")
sys.stdout.flush()
time.sleep(1)
except KeyboardInterrupt:
pass # Ctrl+C: exit quietly, which closes the pipe
Step 2: Set up the writer
The writer program, writer.py, reads the stream from stdin and uploads it to Amazon S3. Steps 2 through 4 build writer.py in one file. Add the following setup code to the top of writer.py:
import signal
import sys
import boto3
MIN_PART_SIZE = 5 * 1024 * 1024 # 5 MiB, the smallest part Amazon S3 accepts
MAX_PART_SIZE = 100 * 1024 * 1024 # largest part your memory budget allows
GROW_EVERY = 2 # demo value; use a larger value in production
bucket, key = sys.argv[1], sys.argv[2]
signal.signal(signal.SIGINT, signal.SIG_IGN) # Ctrl+C stops only the producer
s3 = boto3.client("s3")
upload_id = None # set when the first part is ready
parts = [] # (PartNumber, ETag) pairs for CompleteMultipartUpload
buffer = bytearray()
The writer ignores Ctrl+C so that only the producer stops. The buffer variable is a Python bytearray object, and it grows and shrinks in place.
The MAX_PART_SIZE constant is the memory budget for one part. The GROW_EVERY constant is 2 so that a short run shows the parts growing. For production values, see the Choosing part sizes and handling fast streams section of this article.
Step 3: Add the helpers
The writer uses the following three helpers:
- The next_part_size helper function returns the size of the next part. This example doubles the size every GROW_EVERY parts, up to MAX_PART_SIZE. Replace this function with your own sizing logic, such as a size based on free memory.
- The send_part helper function sends one part with the Boto3 upload_part method. On its first call, it starts the upload with the create_multipart_upload method, which takes no size. Boto3 sets each part's Content-Length from the size of the Body parameter. The helper keeps each ETag with its part number, because the complete_multipart_upload method requires the full, ordered list.
- The cancel_multipart_upload helper function discards the parts already sent if a request fails.
Add the following helpers to writer.py:
def next_part_size():
# Double the part size every GROW_EVERY parts, up to MAX_PART_SIZE.
# Replace this with your own logic, such as a size based on free memory.
size = MIN_PART_SIZE * 2 ** (len(parts) // GROW_EVERY)
return min(size, MAX_PART_SIZE)
def send_part(data):
global upload_id
if upload_id is None: # first part: start the upload, with no size
upload_id = s3.create_multipart_upload(Bucket=bucket, Key=key)["UploadId"]
number = len(parts) + 1
result = s3.upload_part(
Bucket=bucket, Key=key,
PartNumber=number, UploadId=upload_id, Body=data,
)
parts.append({"PartNumber": number, "ETag": result["ETag"]})
print(f"uploaded part {number}: {len(data)} bytes")
def cancel_multipart_upload():
# Cancels the upload and discards any parts already sent.
if upload_id is not None:
s3.abort_multipart_upload(Bucket=bucket, Key=key, UploadId=upload_id)
Step 4: Buffer the stream and upload parts
The loop in this step does the following:
- Reads stdin in 64 KiB chunks into the buffer.
- Sends a part each time the buffer holds next_part_size() bytes.
- Stops when the Python read() method returns no bytes, which marks the end of the stream (end-of-file, or EOF).
- Sends what's left as the last part and then completes the upload.
- Cancels the upload if any request fails.
Add the following code to the end of writer.py:
try:
while True:
chunk = sys.stdin.buffer.read(64 * 1024) # 64 KiB
if not chunk: # EOF: the stream has ended
break
buffer += chunk
while len(buffer) >= (size := next_part_size()): # a full part is ready
send_part(buffer[:size])
del buffer[:size] # drop the sent bytes in place
if buffer: # leftover becomes the last part
send_part(buffer)
if not parts: # empty stream: nothing was started
sys.exit("No data received; nothing was uploaded.")
s3.complete_multipart_upload(
Bucket=bucket, Key=key, UploadId=upload_id,
MultipartUpload={"Parts": parts},
)
print(f"Done: {len(parts)} part(s) uploaded to s3://{bucket}/{key}")
except Exception:
cancel_multipart_upload() # Amazon S3 discards the parts already sent
raise
Boto3 retries temporary errors. If a request still fails, then the writer cancels the upload, because it can't read stdin again to resend the data. If the process crashes first, then the S3 Lifecycle rule deletes the remaining parts.
Step 5: Run the pipeline
Connect the two programs with a pipe, so that the producer's stdout becomes the writer's stdin:
python3 streamer.py | python3 writer.py amzn-s3-demo-bucket streaming-demo.jsonl
Note: Replace amzn-s3-demo-bucket with your bucket name.
Let it run for about 30 seconds, and then press Ctrl+C. The producer exits, and the operating system closes the pipe. The writer ignores the interrupt, so its next read() returns EOF, and it completes the upload.
The output looks similar to the following:
uploaded part 1: 5242880 bytes
uploaded part 2: 5242880 bytes
uploaded part 3: 10485760 bytes
uploaded part 4: 10485760 bytes
uploaded part 5: 20971520 bytes
uploaded part 6: 12161394 bytes
Done: 6 part(s) uploaded to s3://amzn-s3-demo-bucket/streaming-demo.jsonl
Note: Replace amzn-s3-demo-bucket with your bucket name.
The parts double every two parts: 5 MiB, 10 MiB, then 20 MiB. The last part holds what's left.
To check the result, download the first few lines:
aws s3 cp s3://amzn-s3-demo-bucket/streaming-demo.jsonl - | head
Note: Replace amzn-s3-demo-bucket with your bucket name.
Choosing part sizes and handling fast streams
Peak memory is about twice the largest part because the writer holds the buffer and a copy of the part that it's sending.
Because an upload can have at most 10,000 parts, the part sizes set the largest object that you can upload. If every part is the 5 MiB minimum, then an object can reach only about 48.8 GiB. With a GROW_EVERY value of 1,000 and a MAX_PART_SIZE value of 1 GiB, objects can reach about 3.2 TiB. A stream under about 4.9 GiB uses only about 10 MiB of memory. Choose values from your largest expected stream and your memory budget. For more information, see Amazon S3 multipart upload limits.
Streams smaller than 5 MiB also work. If the stream ends before the buffer reaches 5 MiB, then the first part is also the last part. The 5 MiB minimum doesn't apply to the last part, so Amazon S3 accepts it. For a small object of known size, use a single PutObject request instead.
The writer sends parts one at a time. While a part uploads, the writer stops reading stdin, so the pipe fills and the producer's writes wait. When data arrives faster than the writer can upload it, this backpressure slows the producer to the upload rate, and the writer's memory stays fixed. Some sources can't wait, such as a device that drops data that the writer doesn't read in time. For these sources, buffer the data upstream or upload parts in parallel. Each part in flight adds its size to the writer's memory.
Cleanup
To delete the test object, run the following rm command in the AWS CLI:
aws s3 rm s3://amzn-s3-demo-bucket/streaming-demo.jsonl
Note: Replace amzn-s3-demo-bucket with your bucket name.
Important: Parts from an incomplete multipart upload incur storage charges until you or a Lifecycle rule deletes them, and they don't appear in your bucket listing.
To check for incomplete multipart uploads, run the following AWS CLI list-multipart-uploads command:
aws s3api list-multipart-uploads --bucket amzn-s3-demo-bucket
Note: Replace amzn-s3-demo-bucket with your bucket name.
At scale, the S3 Lifecycle rule deletes incomplete multipart uploads automatically. To find storage from incomplete multipart uploads across your buckets, use Amazon S3 Storage Lens.
Conclusion
This article showed how to upload a stream of unknown size to Amazon S3 with multipart upload. Each part declares only its own Content-Length, and Amazon S3 calculates the object's size when it assembles the parts. The writer sizes each part at run time to fit a memory budget and stay under the 10,000-part limit.
If you require control over part size or memory, use the two programs in this article as a starting point. For the complete code, see the sample-s3-stream-unknown-content-length repository on the GitHub website. If you don't require that control, then use a managed client, such as Amazon S3 Transfer Manager or the CRT-based Amazon S3 client.
To learn more about how AWS Support can help you optimize data-intensive workloads on Amazon S3, see AWS Enterprise Support.
Related information
Uploading and copying objects using multipart upload in Amazon S3
Performance guidelines for Amazon S3
About the author
Sri Sarath Kallakuri is a Technical Account Manager (TAM) at AWS supporting enterprise customers in the Energy sector. He focuses on helping organizations architect and optimize large-scale cloud storage, data transfer, and compute workloads.
- Language
- English

Relevant content
asked 2 years ago
asked 5 years ago
asked 3 years ago