How to wait to trigger Lambda function until specific files are uploaded

0

I'm new to AWS and working on Lambda function triggered by S3. I have two json files and need to use the both data to send queue to SQS. For example, two files, "file_a.json" and "file_b.json" are to be uploaded to S3 at the same time.

file_a.json
{"data": "123"}

file_b.json
{"data": "456"}

Then, Lambda will send queue message like ["123", "456"] which is created with the data from both files.

However, S3 invokes Lambda function every time the single file is uploaded and this makes it hard to send queue to SQS as I wrote above. I'm thinking if S3 can wait until the two files are uploaded to trigger Lambda, it can solve the issue. If anyone had faced the similar situation, I would like to know how to resolve this problem. Thanks!

2 Answers
3
Accepted Answer

You will need to put this logic inside your Lambda function. Something like this:

If (my companion file exists in the bucket) then
    if (my file datetime is newer that my companion file) then
        parse files and add data to queue
    end
end
profile pictureAWS
EXPERT
kentrad
answered 2 years ago
profile picture
EXPERT
reviewed 9 months ago
profile pictureAWS
EXPERT
reviewed 2 years ago
0

Hi, short Answer. That is not possible. The S3 Lambda Trigger sends an Event for every Object.

You can send the message to SQS and than trigger the Lambda function via the SQS Trigger. S3 -> SQS -> Lambda -> SQS The SQS Trigger offers a configurable Batch size and Batch window. That can work if you can guarantee that only these two files trigger the S3 Events in the defined Batch window.

AWS
Marco
answered 2 years ago
profile picture
EXPERT
reviewed 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions