S3 Events for burst uploads

1

Hi,

We are planning to ingest approx 2 lakh small pdf files(approx 40 kb) in burst mode from on Premise to Cloud(S3) on daily basis through AWS Datasync. We want to keep track of the files ingested on daily basis and keep updating the processing status as it gets processed.

Can someone suggest the best way to achieve it ? We initially thought of using S3 events to call Lambda to insert record in Dynamo DB table but since it may be a burst of files coming into same S3 bucket it might have throttling issue on Dynamo DB or may hit Lambda limits for the Account.

Please let me know the best way we can implement this.

Thanks in Advance.

Regards, Dhaval Mehta

4개 답변
2
수락된 답변

There are multiple ways to achieve this. But it really depends what kind of status you want to track.

For real timenotifications within a short duration, you can take advantage of Event notification from object upload and pair it with SNS/SQS/Lambda and use some sort of Database or a text/csv file to keep track of objects uploaded. This will require code.

If the tracking can wait for 24+ hours, then S3 inventory is a better choice. It's very easy to configure without any code.

AWS
답변함 2년 전
0

Decouple the process by sending the S3 notification to a SQS queue. https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-enable-disable-notification-intro.html

Then have your Lambda function poll the queue. https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

profile pictureAWS
전문가
kentrad
답변함 2년 전
0
profile picture
답변함 2년 전
0

I would say that even going with Lambda notification should be ok for you if the lambda is only saving to DynamoDB because it should be super fast. It's not so easy to reach the account limit. But it is possible, so you can notify SQS instead of Lambda and trigger Lambda from SQS but using batch. For standard SQS it can be maximum 10000 items in one batch, so you will call Lambda much less often.

profile picture
MG
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠