Looking for different ways to Orchestrate Glue ETL jobs

0

I have a scenario where I have multiple glue ETL jobs which are interdependent, and they have some logical order to follow. I am looking for the best possible approaches within aws solutions to trigger a group of Glue jobs, based on the success/failure state of a different group of Glue jobs, i.e., setting up a combination of series and parallel execution of jobs under one entity which can then be reused in another such entity to avoid building the whole flow again (like we use a shell script to group and conditionally orchestrate python scripts). A GUI visual to represent the dataflows will be an added advantage.

I have tried and tested the Workflow feature within Glue to simulate this requirement, I was able to create the grouping of jobs based on triggers, but the major drawback was that I could not call/invoke existing workflows into a bigger workflow (like a parent WF which can fire the end-to-end ETL), thereby requiring me to build the whole flow again each and every time.

I have knowledge on SAP BODS ETL (if we need to draw comparisons), requesting experts' views to address this requirement. Thanks in advance!

1 réponse
2

I would recommend looking into Step Functions, a serverless orchestration service, which lets you orchestrate over 200 AWS services, including Glue jobs.

profile pictureAWS
EXPERT
Uri
répondu il y a un an

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions