Glue Job - S3 to S3

Hi Team,

I am working on Glue to job to copy/move file from one bucket to another bucket. Could you please help me with your thoughts

Using Python how to copy/move the unzipped file to target bucket if the source file is tar type ?
Using Python how to use stream and multiple part upload approach to handle large file
As per my understanding we can write file to target bucket using either assume writer or secret &access key. Would like know if any other methods are available other than these 2 methods ?

Note : The source and target bucket may be in same or different AWS account, same or different region

Thank you in advance !

secondabhi_aws EXPERT
10 days ago
Can you please clarify little more, why are you considering glue for file transfer between s3 buckets, instead of lambda. Are there too many files to copy, files sizes are big, which can't be completed within lambda timeout(15 minutes). Are there any transformation, you'll be doing on source file before copying it to target bucket. Why aren't you considering native s3 features for file transfers(is it for specific files in a certain workflow, if so then I can understand not using native s3 features).
Bharath
10 days ago
Thank you @secondabhi_aws.

1 . We are expecting many files and files size may be big which can't completed within 15 mins. The solution should be scalable.
2. The source and target bucket may be vary from team to team, and this solution should be generic one(this can be maintain as configurable) 3. The files in source bucket may be compressed format(zip or tar), and based on the project team requirement it may be unzip and loaded in target bucket.

Topics

Relevant content