Running into much stricter Rate Limits than described in S3 documentation

0

I am trying to selectively clear out an S3 bucket with roughly 3 billion objects based on some custom made inventory manifest files. I am using Lambda to run a function that will read a manifest file (of up to 25000 files), and then make decisions to copy and delete the appropriate files (~5% will be copies and ~85% will be deletes). These lambdas are running in parallel but will always be working on different S3 prefixes. The lambda function scales up the calls to DeleteObjects from 50 objects per call up to the max of 1000 objects per call.

However, I am consistently running into rate limiting errors (SlowDown) during these delete calls. The S3 docs specify a rate limit of 3500 COPIES/DELETES per second per prefix, but at max each prefix is experiencing 1000 deletes per second and usually much less. It also has retry logic to back off to up to a minute sleep before making another request 10 times, but still the requests will fail quite often (~5% of calls), and I cannot increase the exponential backoff by much because the lambda will time out in 15 minutes. Additionally, the number of lambdas I can run in parallel is capped at about 100 before the error rate spikes dramatically.

Can I get some more insight into the S3 rate limits and why I cannot come even close to the 3500 per prefix per second rate limit? Or potentially a solution to increase overall throughput while avoiding rate limiting errors?

wil
asked a year ago552 views
1 Answer
1
Accepted Answer

I assume you are referring to this documentation.

It also states, "The scaling, in the case of both read and write operations, happens gradually and is not instantaneous. While Amazon S3 is scaling to your new higher request rate, you may see some 503 (Slow Down) errors. These errors will dissipate when the scaling is complete. For more information about creating and using prefixes, see Organizing objects using prefixes."

You are most likely driving the automatic scaling of S3 and would need to maintain the rates to accomplish the top of the limits.

profile pictureAWS
answered a year ago
profile picture
EXPERT
reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions