S3 upload files and wait 5 minute before processing with lamba

Question

Hi,

we have a use case that a user can upload multiple files on S3 and we don't know how many will be uploaded and neither we can have an event of finish upload but we want to process at batch of 5 minutes to speed up the processing avoiding using a queue to process files at arrival

for example:
- the user upload 5 file
- upon arrival of first file start star a timer of 5 minutes to wait possible other files
- exceeded the 5 minutes start the processing of arrived files

We think about using SQS with batch window and batch size but we cannot guarantee, as we thought, that the time window will be respected. The second option would be use dynamoDB to store file arrival and build a logic to process file.

Any other idea or possibility using SQS or eventbridge without writing custom code?

Answer

Hi Paolo - You can use [Amazon SQS delay queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html) to ensure 5 minutes delay. So flow could be like this -

File uploaded to S3 --> Event Bridge rule to push the messages into SQS delay queue --> Message is delivered after the delay.

This way you will be sure that there's a 5 minutes delay before you receive the message, so when your code runs, you can just look for the files present in S3 at that time.

Answer

There is no single service that will give you want you want. I think your best option is to catch the file upload event. If the upload belongs to a new session, create an EventBridge schedule that will fire in 5 minutes. If the file belongs to an existing session, discard the event, or maybe extend the timer, if it makes sense.

You should use something like DynamoDB to track the sessions, and if you want, you can also add the uploaded files to the DDB table so that later, when the timer expires, you will already have the list of files.

S3 upload files and wait 5 minute before processing with lamba

Relevant content