1 Risposta
- Più recenti
- Maggior numero di voti
- Maggior numero di commenti
0
Hi Paolo,
I understand you would like to accomplish this ETL job with the visual editor as much as possible. Unfortunately at the time, there is not a native purge "Action" which can be used to clear out that S3 staging area. However, the "Custom Transform" would be a good fit and the GlueContext includes two methods which you can use to easily purge the staging data: purge_table and purge_s3_path. This could all be done within that job and I don't think the Step Functions state machine is necessary
Let me know if you have any other questions
Loc D., AWS ProServe
con risposta un anno fa
Contenuto pertinente
- AWS UFFICIALEAggiornata 2 anni fa
- AWS UFFICIALEAggiornata 2 anni fa
- AWS UFFICIALEAggiornata un anno fa
Hi, yes was my first go but since the users have no coding experience I looked at the step function solution. I also thought to create a single glue job with a custom script to purge s3 paths, this way the job could be used inside a glue workflow but it's not possible to set dynamic parameters as input other than mapping at workflow level and could be the case that the workflow manage many paths to be purged with a poor manageability on the long term. Thanks for the feedback