1 réponse
- Le plus récent
- Le plus de votes
- La plupart des commentaires
0
Hi Paolo,
I understand you would like to accomplish this ETL job with the visual editor as much as possible. Unfortunately at the time, there is not a native purge "Action" which can be used to clear out that S3 staging area. However, the "Custom Transform" would be a good fit and the GlueContext includes two methods which you can use to easily purge the staging data: purge_table and purge_s3_path. This could all be done within that job and I don't think the Step Functions state machine is necessary
Let me know if you have any other questions
Loc D., AWS ProServe
répondu il y a un an
Contenus pertinents
- demandé il y a un an
- demandé il y a 7 mois
- demandé il y a un an
- demandé il y a un an
- AWS OFFICIELA mis à jour il y a un an
- Comment puis-je résoudre les erreurs de connexion à AWS Marketplace dans mes tâches ETL d’AWS Glue ?AWS OFFICIELA mis à jour il y a 3 mois
Hi, yes was my first go but since the users have no coding experience I looked at the step function solution. I also thought to create a single glue job with a custom script to purge s3 paths, this way the job could be used inside a glue workflow but it's not possible to set dynamic parameters as input other than mapping at workflow level and could be the case that the workflow manage many paths to be purged with a poor manageability on the long term. Thanks for the feedback