Looking for any process locking or latest file processing with msk and lambda


Hi, we have a use case where some files will be placed into s3 and an event will be placed to MSK to trigger a lambda event to process the file. Each file will correspond to a client identifier and we can have multiple files for one client identifier with a different processing timestamp in it. As lambda will be triggered for each file, is there a mechanism to check if any lambda started processing for the same client identifier file and make sure to process only latest files based on processing timestamp check? Process Locking: If we can create a locking mechanism with a key so that while processing any file, a check will happen whether any other file with same client identifier picked up and is in process so that until that process is completed, latest process should wait. Please let me know if you need more information on it

1 Answer

You'll need some sort of external (i.e.outside of Lambda) data store to keep track of what is currently being processed.

If it were me, I'd create a DynamoDB table and store a record in the table for each document that has started processing. I'd also check at the beginning of processing to see if there was already a record for the document that is about to be processed. At that point, also check the timestamp. At the end of processing, delete the record.

There is a possibility here of a race condition where one file is uploaded and has started processing and then another file is uploaded while the first is still in progress. I'm not sure how you would want to handle that.

Finally, in the event that processing failed I would put a TTL on the DynamoDB record so that it automatically is deleted and unblocks future processing.

I suspect there are some other requirements that will come out of this - I'd encourage you to reach out to your local AWS Solutions Architect to discuss what the best way forward is.

profile picture
answered 2 months ago
profile picture
reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions