Looking for different ways to Orchestrate Glue ETL jobs

0

I have a scenario where I have multiple glue ETL jobs which are interdependent, and they have some logical order to follow. I am looking for the best possible approaches within aws solutions to trigger a group of Glue jobs, based on the success/failure state of a different group of Glue jobs, i.e., setting up a combination of series and parallel execution of jobs under one entity which can then be reused in another such entity to avoid building the whole flow again (like we use a shell script to group and conditionally orchestrate python scripts). A GUI visual to represent the dataflows will be an added advantage.

I have tried and tested the Workflow feature within Glue to simulate this requirement, I was able to create the grouping of jobs based on triggers, but the major drawback was that I could not call/invoke existing workflows into a bigger workflow (like a parent WF which can fire the end-to-end ETL), thereby requiring me to build the whole flow again each and every time.

I have knowledge on SAP BODS ETL (if we need to draw comparisons), requesting experts' views to address this requirement. Thanks in advance!

已提問 1 年前檢視次數 655 次
1 個回答
2

I would recommend looking into Step Functions, a serverless orchestration service, which lets you orchestrate over 200 AWS services, including Glue jobs.

profile pictureAWS
專家
Uri
已回答 1 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南