For Amazon S3 request rates, what's the difference between prefixes and nested folders? How many prefixes can I have in an S3 bucket?
For Amazon Simple Storage Service (Amazon S3) request rates, what's the difference between key prefixes and nested folders? How many prefixes can I have in an S3 bucket?
Resolution
Prefixes
A key prefix is a string of characters that can be the complete path in front of the object name (including the bucket name). For example, if an object (123.txt) is stored as BucketName/Project/WordFiles/123.txt, the prefix might be “BucketName/Project/WordFiles/123.txt”. The prefix can be any length, including the entire object key name.
If the 123.txt file is saved in a bucket without a specified path, Amazon S3 automatically adjusts the prefix value according to the request rate. Partitions can be automatically formed at any point in the prefix string.
A partitioned prefix in a bucket can support 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second. There is no limit to the number of prefixes that you can have in a bucket. However, be aware that a spike in the request rate can cause throttling.
Note: In Amazon S3, there are no partitions for keys or objects. Partitions exist only at the prefix level, and not at the object level. For more information about using prefixes in Amazon S3, see Organizing objects using prefixes.
Folders
In Amazon S3, folders are used to group objects and organize files. Unlike a traditional file system, Amazon S3 doesn't use hierarchy to organize its objects and files. Amazon S3 console supports the folder concept only as a means of grouping (and displaying) objects.
More specifically, a folder is the value between the two "/" characters. For example, if a file is stored as BucketName/Project/WordFiles/123.txt, the file path indicates that there is a folder ("Project") and subfolder ("WordFiles"). Both "Project" and "WordFiles" are considered to be folders. If the 123.txt file is saved in a bucket without a specified path, then no folders are used to store the file.
Note: The folder structure might not indicate any partitioned prefixes that support request rates.
Difference between prefixes and folders
The difference between a prefix and a folder is the significance of the "/" character. For folders, the "/" character signifies a subfolder or object name. For prefixes, "/" is just another character. The "/" does not indicate a partition placement.
Note: The folder structure only applies to the Amazon S3 console. For more information, see Organizing objects in the Amazon S3 console using folders.
Related information
AWS re:Invent 2018: Best practices for Amazon S3 and Amazon S3 Glacier
This is still quite confusing I think.
Amazon S3 automatically adjusts the prefix value according to the request rate.
So as an end-user I have no way of impacting prefixing at all, because it's all automatic?
A partitioned prefix in a bucket can support 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second.
And by extension it's impossible to know what the actual limit is? Because this is a lower bound that is multiplied by a prefix value that is managed by AWS and invisible to us as the end user?
For prefixes, "/" is just another character. The "/" does not indicate a partition placement.
This also implies that the prefix partitioning can happen anywhere in the key name?
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
In the video on youtube, you said the prefix is "BucketName/Project/WordFiles/", but in this post, you said the prefix is “BucketName/Project/WordFiles/123.txt”. Which is correct?
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
It's been 3 months and you still haven't updated the article or even properly acknowledged the questions. "Thank you, we will review, etc." is not an adequate response because it is generic. You should be answering the specific questions with specific answers if you expect folks to actually get help from this knowledge-center.
Relevant content
- asked a month ago
- Accepted Answerasked 3 years ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 10 months ago
