move files between s3 buckets upon complete load

Question

Hi team,

I have an s3 bucket that accepts 20 CSV files through SFTP transfer, I want to move those 20 files to another s3 bucket for treatment only and only if ALL 20 files have been loaded on the first bucket.

what is the best option to do that transfer under the condition that all 20 files are completely loaded in the first bucket to trigger the transfer to the second bucket?

If I use lambda that listens for s3 PutObject Event, there is no risk of parallel invocation if 20 events are generated at the same time?

Thank you.

Accepted Answer

I would use a Lambda function triggered by any create action that had logic like this:

```
get count of objects in bucket
if count = 20 then
   get create datetime for triggered object
   get create datetime of objects in the bucket
   loop through datetimes
      find the oldest create datetime
  if triggered object datetime = oldest datetime
     move the files
  end
end
```

Answer

Why do you have to wait at all? Why not move them one at a time? Also, this action is idempotent. If the file was already moved, it can return success.

move files between s3 buckets upon complete load

Relevant content