Receiving S3 503 slow down responses

Question

We have an application that stores a large volume of data in one S3 bucket. Lately, we have started receiving 503 slow-down error messages. Reading the docs, it seems like the API is related to the Prefix and Partitions that S3 creates internally based on the Folder structure.

Our current structure is /uploads/receipt_image_v2// ... I wonder if we should do something different? We have millions of UUID and 1-2 images per UUID in the S3 bucket.

I can't find any way to see how many partitions there are for our S3 bucket.

venky81 · Accepted Answer

Your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. You can work with customer support to pre partition the bucket based on your folder naming structure. Last time we did it we had to create a new bucket & work with support to pre partition it & copy data from the old bucket to new one.

Nuno_Q · Answer

Hi
have a look at https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/ to troubleshoot HTTP 500 or 503 error from Amazon S3.

Also have a look at:

https://aws.amazon.com/premiumsupport/knowledge-center/s3-503-within-request-rate-prefix/

If there is a fast spike in the request rate for objects in a prefix, Amazon S3 might return 503 Slow Down errors while it scales in the background to handle the increased request rate. To avoid these errors, you can configure your application to gradually increase the request rate and retry failed requests using an exponential backoff algorithm [1].

[1] https://docs.aws.amazon.com/general/latest/gr/api-retries.html

Receiving S3 503 slow down responses

Relevant content