S3 Partitioned prefixes

Question

Greetings AWS Community,

I'm currently navigating through a Skillbuilder course and find myself puzzled by a particular concept: Partitioned prefixes. In the course materials, it's mentioned:

"Amazon S3 supports up to 3,500 PUT/POST/DELETE and 5,500 GET transactions per second (TPS) per partitioned prefix. Partitions are logical entities that Amazon S3 uses internally to index your object keys. Initially, object keys in a bucket reside on a single partition.

As Amazon S3 detects sustained request rates that exceed a single partition's capacity, it creates a new partition per prefix in your bucket. As the throughput requirements for your workloads grow, the partitioned prefixes scale accordingly. There are no limits to the number of prefixes Amazon S3 supports."

I'm struggling to grasp the concept of partitioned prefixes. What exactly is a partitioned prefix? Despite my attempts to research, I couldn't find a clear explanation. The closest I got was this insight from Amazon Q:

"When objects are uploaded to S3, they are stored across different partitions based on the prefix. So while the prefix may be the same for all objects, S3 is able to distribute the load across partitions automatically based on the object keys."

Furthermore, how does Amazon initiate the creation of these new partitions as our workload increases? Does S3 handle this process automatically, or are there specific settings we need to enable? How does S3 determine the names of the partitions?

Your insights and experiences on this matter would be greatly appreciated.

Thank you!

Antonio_Lagrotteria · Accepted Answer

As pointed out here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html

> You can use prefixes to organize the data that you store in Amazon S3 buckets. A prefix is a string of characters at the beginning of the object key name.

Partitioned prefixes are "partitioned" because they divide and organize your data to create a hierarchical structure, which then can be used for indexing and retrieval purposes.

As per "behind the scenes" tricks, as pointed out here by an AWS employee: https://repost.aws/knowledge-center/s3-object-key-naming-pattern, there is some internal optimization that S3 performs.

> Additionally, it's important to note that auto partitioning happens behind the scenes and involves S3 services monitors services that run automatically and the process can take from 30 to 60 minutes. Should the customer choose to do so, they can pre-petition using AWS support.

S3 Partitioned prefixes

Relevant content