Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.
What's the difference between prefixes and nested folders for Amazon S3 request rates?
I want to understand the effect of prefixes and nested folders on Amazon Simple Storage Service (Amazon S3) request rates.
Resolution
Use prefixes to handle increases in request rates for your Amazon S3 buckets. On the Amazon S3 console, prefixes are called folders and might not show partitioned prefixes that support request rates. For more information, see Organizing, listing, and working with your objects.
Prefixes
A key prefix is a string of characters at the beginning of the object name. For example, if you store an object as BucketName/Project/WordFiles/123.txt, then the prefix is Project/WordFiles/ and the key name is Project/WordFiles/123.txt. Prefix length can be up to the maximum 1,024 byte-length of the object key name.
If the 123.txt file is in a bucket without a specified path, then Amazon S3 automatically adjusts the prefix value based on the request rate. Amazon S3 can automatically create partitions at any point in the prefix string.
A partitioned prefix in a bucket can support 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second. There's no limit to the number of prefixes that you can have in a bucket. However, a spike in the request rate might cause throttling.
Note: There are no partitions for keys or objects. Partitions exist only at the prefix level, not at the object level. For more information about how to use prefixes in Amazon S3, see Organizing objects using prefixes.
Folders
A folder is the value between two slash (/) characters. For example, if you store a file as BucketName/Project/WordFiles/123.txt, then both Project and WordFiles are folders. The folder is Project, and the subfolder is WordFiles. If you save the 123.txt file in a bucket without a specified path, then Amazon S3 doesn't use folders to store the file.
A folder groups objects and organizes files. Amazon S3 doesn't use a hierarchy to organize objects and files. The Amazon S3 console supports the folder concept only to group and display objects.
When you create a folder, Amazon S3 creates a 0-byte object with a key that it sets to the provided folder name. The key includes a trailing forward slash (/) character.
- Topics
- Storage
- Language
- English
This is still quite confusing I think.
Amazon S3 automatically adjusts the prefix value according to the request rate.
So as an end-user I have no way of impacting prefixing at all, because it's all automatic?
A partitioned prefix in a bucket can support 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second.
And by extension it's impossible to know what the actual limit is? Because this is a lower bound that is multiplied by a prefix value that is managed by AWS and invisible to us as the end user?
For prefixes, "/" is just another character. The "/" does not indicate a partition placement.
This also implies that the prefix partitioning can happen anywhere in the key name?
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
In the video on youtube, you said the prefix is "BucketName/Project/WordFiles/", but in this post, you said the prefix is “BucketName/Project/WordFiles/123.txt”. Which is correct?
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
It's been 3 months and you still haven't updated the article or even properly acknowledged the questions. "Thank you, we will review, etc." is not an adequate response because it is generic. You should be answering the specific questions with specific answers if you expect folks to actually get help from this knowledge-center.
This post is quite unclear and confusing. It also points to the 2018 video which is out of date. Probably best to just delete this post and point to the current s3 performance whitepaper at: https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/introduction.html
This does not clarify the difference between prefixes and folders. If we use "/" as a delimiter, does that mean it will create folders? What if the delimiter is "-"? Then folders won't be created? But you can still get the benefits of prefixes, or no?
Every example for prefixes all use "/" as if they are creating folders so I do not understand if prefixes and folders are just interchangeable or if something different will happen with a different delimiter.
I am frustrated because this question came up on an exam and I got it incorrect because I chose the folder option and it wanted the prefix option which to me sound like the exact same thing.
Relevant content
- asked a year ago
- asked 2 years ago
- asked 2 years ago
