move files between s3 buckets upon complete load
Hi team,
I have an s3 bucket that accepts 20 CSV files through SFTP transfer, I want to move those 20 files to another s3 bucket for treatment only and only if ALL 20 files have been loaded on the first bucket.
what is the best option to do that transfer under the condition that all 20 files are completely loaded in the first bucket to trigger the transfer to the second bucket?
If I use lambda that listens for s3 PutObject Event, there is no risk of parallel invocation if 20 events are generated at the same time?
Thank you.
I would use a Lambda function triggered by any create action that had logic like this:
get count of objects in bucket
if count = 20 then
get create datetime for triggered object
get create datetime of objects in the bucket
loop through datetimes
find the oldest create datetime
if triggered object datetime = oldest datetime
move the files
end
end
Why do you have to wait at all? Why not move them one at a time? Also, this action is idempotent. If the file was already moved, it can return success.
Relevant questions
S3 Glacier (2013) Flexible running without Bucket, want to move to Glacier Deep Archive
asked a month agoMove S3 buckets to different account
Accepted AnswerHow to determine if an object is encrypted with a "regular" S3-SSE KMS key, or an S3 Bucket Key with S3 Inventory?
Accepted Answerasked a year agobest practice to move ETL reated files
asked a month agomove files between s3 buckets upon complete load
Accepted Answerasked a month agoOptions to accelerate s3 copy of 4TB worth files between S3 buckets in same region
Accepted Answerasked 2 years agoGlue ETL job write part-r-00 files to same bucket as my input. Any way to change this?
Accepted Answerasked 2 months agoCSV crawler tables name
asked 4 months agoHow can we set s3 lifecycle rule to delete the files form S3 bucket!
asked a month agoMove files from S3 Bucket back to Site
asked 2 months ago