how to handle file processing with Lambda

0

Hello I've files which are constantly created on S3 with a standard format, fixed lenght; say it's AAA-YYY-ZZZ. Depending on the values assumed by YYY there are different lambda to be invoked to process the file (these lambdas are already implemented in Java, not easy to refactor or change).

From what we've seen, we have two options:

  1. set up a filter based on the prefix or suffix of the file name. However, S3 does not support filters based on substrings other than prefixes or suffixes. I could use AAA or ZZZ value, but not YYY. So this doesn't seem to be an option.
  2. Use an intermediary lambda function, that will select and call the second lambda.

Is there any other way we can do this filtering/triggering? Or which is the best approach to architect this ?

AWS
Diego_C
asked 4 days ago221 views
3 Answers
2

Hello.

I thought option 2 was effective if it was not possible to refactor the currently existing Lambda code.
I think the problem can be solved by calling Lambda with an event trigger when the file is created in S3 and calling the existing Lambda if it matches "YYY".

profile picture
EXPERT
answered 4 days ago
profile picture
EXPERT
reviewed 4 days ago
profile pictureAWS
EXPERT
iBehr
reviewed 4 days ago
2

You can send the S3 events to EventBridge and there you can create different filters using wildcards, each filter will trigger a different function.

Look at this blog post that describes wildcards, which includes a very similar example to yours.

profile pictureAWS
EXPERT
Uri
answered 4 days ago
profile picture
EXPERT
reviewed 4 days ago
profile pictureAWS
EXPERT
iBehr
reviewed 4 days ago
0

You can also leverage the S3 Object Tags. If it's possible for the client app to add tag each time an object is created in S3 bucket, e.g., { "YYY": "<value_of_yyy>" }, then you can get your (intermediary) Lambda function (trigger) to evaluate the value of this tag and call the corresponding (Java) Lambda function to process the object based on the value of your YYY tag.

answered 4 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions