By using AWS re:Post, you agree to the Terms of Use
/move files between s3 buckets upon complete load/

move files between s3 buckets upon complete load


Hi team,

I have an s3 bucket that accepts 20 CSV files through SFTP transfer, I want to move those 20 files to another s3 bucket for treatment only and only if ALL 20 files have been loaded on the first bucket.

what is the best option to do that transfer under the condition that all 20 files are completely loaded in the first bucket to trigger the transfer to the second bucket?

If I use lambda that listens for s3 PutObject Event, there is no risk of parallel invocation if 20 events are generated at the same time?

Thank you.

2 Answers
Accepted Answer

I would use a Lambda function triggered by any create action that had logic like this:

get count of objects in bucket
if count = 20 then
   get create datetime for triggered object
   get create datetime of objects in the bucket
   loop through datetimes
      find the oldest create datetime
  if triggered object datetime = oldest datetime
     move the files
answered a month ago

Why do you have to wait at all? Why not move them one at a time? Also, this action is idempotent. If the file was already moved, it can return success.

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions