best practice to move ETL reated files

0

Hi Team,

I have a glue job that reads CSV files from S3 and injects data to Mysql RDS,

I want to move all files from s3 to glacier once they're treated by ETL glue.

I would like to know what is the best architecture to move CSV files from s3 to glacier upon glue job success?

is it better to do it from within the glue script at the end of the script?

or

to do it through another process?

also, any code example will be helpful :)

Thank you :)

1 Antwort
0
  1. If you have a table created in Glue catalog and to directly transition the storage class of files related to this table, you can use transition_table API from within the Glue job.
  2. If you do not have a table created, you can use transition_s3_path API to directly transition the storage class of files by specifying an s3 path from within the Glue job.
AWS
SUPPORT-TECHNIKER
Manu_G
beantwortet vor 2 Jahren
AWS
EXPERTE
überprüft vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen