Invoke Lambda functions from S3 uploads at high scale

Question

Customers who wants to upload files to S3, trigger Lambda functions, and write the file name in a database. This scenario can cause a 10,000 uploads per second.

There is no 100% guarantee that for every file uploaded, a Lambda function will be invoked.

What would be a good way to deal with the peaks and write the file names to a database?

Accepted Answer

Recently S3 changed their notification mechanism and it is now at least once delivery of events. If you want to make sure that all events are handled, send them to SQS and trigger a Lambda from the queue. in addition to notifications you can compare the S3 Inventory to what's in the database to catch missed events.

Invoke Lambda functions from S3 uploads at high scale

Relevanter Inhalt