Looking for any process locking or latest file processing with msk and lambda

0

Hi, we have a use case where some files will be placed into s3 and an event will be placed to MSK to trigger a lambda event to process the file. Each file will correspond to a client identifier and we can have multiple files for one client identifier with a different processing timestamp in it. As lambda will be triggered for each file, is there a mechanism to check if any lambda started processing for the same client identifier file and make sure to process only latest files based on processing timestamp check? Process Locking: If we can create a locking mechanism with a key so that while processing any file, a check will happen whether any other file with same client identifier picked up and is in process so that until that process is completed, latest process should wait. Please let me know if you need more information on it

1개 답변
2

You'll need some sort of external (i.e.outside of Lambda) data store to keep track of what is currently being processed.

If it were me, I'd create a DynamoDB table and store a record in the table for each document that has started processing. I'd also check at the beginning of processing to see if there was already a record for the document that is about to be processed. At that point, also check the timestamp. At the end of processing, delete the record.

There is a possibility here of a race condition where one file is uploaded and has started processing and then another file is uploaded while the first is still in progress. I'm not sure how you would want to handle that.

Finally, in the event that processing failed I would put a TTL on the DynamoDB record so that it automatically is deleted and unblocks future processing.

I suspect there are some other requirements that will come out of this - I'd encourage you to reach out to your local AWS Solutions Architect to discuss what the best way forward is.

profile pictureAWS
전문가
답변함 2년 전
profile pictureAWS
전문가
Chris_G
검토됨 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠