best practice to move ETL reated files
0
Hi Team,
I have a glue job that reads CSV files from S3 and injects data to Mysql RDS,
I want to move all files from s3 to glacier once they're treated by ETL glue.
I would like to know what is the best architecture to move CSV files from s3 to glacier upon glue job success?
is it better to do it from within the glue script at the end of the script?
or
to do it through another process?
also, any code example will be helpful :)
Thank you :)
asked 2 months ago19 views
1 Answers
0
- If you have a table created in Glue catalog and to directly transition the storage class of files related to this table, you can use transition_table API from within the Glue job.
- If you do not have a table created, you can use transition_s3_path API to directly transition the storage class of files by specifying an s3 path from within the Glue job.
Relevant questions
best practice to move ETL reated files
asked 2 months agoCan AWS Glue read data from different SQL Server table, generate csv files and zipping it to S3?
Accepted Answerasked 3 months agoHow could we have Glue to get data from csv as String?
Accepted Answerasked 17 days agoMove files from S3 to FSx for Windows
Accepted Answerasked 2 years agoCannot see files in S3/Glacier when using Synology Glacier app for backup
asked 4 days agoMove files from S3 Bucket back to Site
asked 2 months agoGlue ETL job write part-r-00 files to same bucket as my input. Any way to change this?
Accepted Answerasked 2 months agoHow to read from CSV files in S3 that have headers?
Accepted Answerasked 4 years agomove files between s3 buckets upon complete load
Accepted Answerasked 2 months agoS3 Glacier (2013) Flexible running without Bucket, want to move to Glacier Deep Archive
asked a month ago