S3 performance problems: How can i achieve a s3 request rate above the limit of 3500 PUT per second with multiple prefixes?

Question

According to the [official document](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html), I try to scale write operations by writing to multiple prefixes, i got nothing, but the same limits(3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second). The official document  seems misleading or incorrect.

Q: How can i  achieve a s3 request rate above the limit of 3500 PUT per second? Or, how can i make the prefix partitioned ?

PS：I got a solution in[ the link ](https://aws.amazon.com/premiumsupport/knowledge-center/s3-request-limit-avoid-throttling/), but how to "gradually scale up" as described in it ?

Answer

Throughput on S3 is based on the prefix of the object. The prefix is the initial part of the object name, so for an object with the name upload/data/incoming/file.txt, the prefix is upload/data/incoming/. Each prefix will receive upto the limits you list above, so to increase the 'bucket' throughput, ensure you use unique prefixes.

One thing to remember is that whilst we often think of the components in the prefix as folders, they're not really, so upload is a different prefix to upload/data.

One thing to remember is that whilst we often think of the components in the prefix as folders, they're not really, so `upload` is a different prefix to `upload/data`.

S3 performance problems: How can i achieve a s3 request rate above the limit of 3500 PUT per second with multiple prefixes?

相關內容