Looking for different ways to Orchestrate Glue ETL jobs

0

I have a scenario where I have multiple glue ETL jobs which are interdependent, and they have some logical order to follow. I am looking for the best possible approaches within aws solutions to trigger a group of Glue jobs, based on the success/failure state of a different group of Glue jobs, i.e., setting up a combination of series and parallel execution of jobs under one entity which can then be reused in another such entity to avoid building the whole flow again (like we use a shell script to group and conditionally orchestrate python scripts). A GUI visual to represent the dataflows will be an added advantage.

I have tried and tested the Workflow feature within Glue to simulate this requirement, I was able to create the grouping of jobs based on triggers, but the major drawback was that I could not call/invoke existing workflows into a bigger workflow (like a parent WF which can fire the end-to-end ETL), thereby requiring me to build the whole flow again each and every time.

I have knowledge on SAP BODS ETL (if we need to draw comparisons), requesting experts' views to address this requirement. Thanks in advance!

已提问 1 年前703 查看次数
1 回答
2

I would recommend looking into Step Functions, a serverless orchestration service, which lets you orchestrate over 200 AWS services, including Glue jobs.

profile pictureAWS
专家
Uri
已回答 1 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则