By using AWS re:Post, you agree to the Terms of Use
/ETL Workflow Orchestration Step functions and/or Glue Workflows??/

ETL Workflow Orchestration Step functions and/or Glue Workflows??


IHAC who's doing a low level design on their data lake. they want to use all AWS native services where possible. they have a question on ETL orchestration best practices on AWS. They were looking at Step functions but since Glue Workflow is available since Jun 2019 they were wondering which to use or a combo. Of course they are looking for the easy button. here's their primary requirements.

  1. ETL orchestration - step functions vs. Glue Workflow
    1. ~150 sources all sending files various times
    2. Source systems have limits on concurrency that scheduling tool must support
    3. example max 10 concurrent jobs for ACME source - job scheduling tool should pole and submit jobs keeping 10 active jobs but no more than 10
    4. ETL jobs should be built off of parameterized template where they pass in parameters like source, table name date and the job auto builds vs. having to maintain library of jobs/scripts per source/table. want this to be dynamically built
    5. Alerts on ETL processing
      1. Cloudwatch alert to SNS topics to etl teams on failures
      2. Cloudwatch alert to SNS for business users(loads complete)
      3. etc
    6. Support downstream jobs/etl example load file A & Load file B once completed for the day should launch load file C etc
1 Answers
Accepted Answer

Hey Dave,

It sounds like a perfect use case for Glue especially because the quantities and the concurrency is not too masive.

Good luck! Ido

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions