move files between s3 buckets upon complete load

0

Hi team,

I have an s3 bucket that accepts 20 CSV files through SFTP transfer, I want to move those 20 files to another s3 bucket for treatment only and only if ALL 20 files have been loaded on the first bucket.

what is the best option to do that transfer under the condition that all 20 files are completely loaded in the first bucket to trigger the transfer to the second bucket?

If I use lambda that listens for s3 PutObject Event, there is no risk of parallel invocation if 20 events are generated at the same time?

Thank you.

2回答
0
承認された回答

I would use a Lambda function triggered by any create action that had logic like this:

get count of objects in bucket
if count = 20 then
   get create datetime for triggered object
   get create datetime of objects in the bucket
   loop through datetimes
      find the oldest create datetime
  if triggered object datetime = oldest datetime
     move the files
  end
end
profile pictureAWS
エキスパート
kentrad
回答済み 2年前
profile picture
エキスパート
レビュー済み 1ヶ月前
0

Why do you have to wait at all? Why not move them one at a time? Also, this action is idempotent. If the file was already moved, it can return success.

profile pictureAWS
回答済み 2年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ