ETL Workflow Orchestration Step functions and/or Glue Workflows??

0

IHAC who's doing a low level design on their data lake. they want to use all AWS native services where possible. they have a question on ETL orchestration best practices on AWS. They were looking at Step functions but since Glue Workflow is available since Jun 2019 they were wondering which to use or a combo. Of course they are looking for the easy button. here's their primary requirements.

  1. ETL orchestration - step functions vs. Glue Workflow
    1. ~150 sources all sending files various times
    2. Source systems have limits on concurrency that scheduling tool must support
    3. example max 10 concurrent jobs for ACME source - job scheduling tool should pole and submit jobs keeping 10 active jobs but no more than 10
    4. ETL jobs should be built off of parameterized template where they pass in parameters like source, table name date and the job auto builds vs. having to maintain library of jobs/scripts per source/table. want this to be dynamically built
    5. Alerts on ETL processing
      1. Cloudwatch alert to SNS topics to etl teams on failures
      2. Cloudwatch alert to SNS for business users(loads complete)
      3. etc
    6. Support downstream jobs/etl example load file A & Load file B once completed for the day should launch load file C etc
1 Answer
0
Accepted Answer

Hey Dave,

It sounds like a perfect use case for Glue especially because the quantities and the concurrency is not too masive.

Good luck! Ido

answered 5 years ago
  • Hi Dave, Thank you for your response. I still have some doubts regarding the parameters you mentioned in the first point, and I believe you can assist me with that. Currently, I am using a Glue Workflow to process my data, which is triggered by an Event using EventBridge. This workflow starts with a Glue Job that fetches the last file uploaded to the S3 bucket being monitored, which triggers the event. However, I'm unsure how to adapt my Glue Job to get the exact file that triggered the workflow. I am concerned about how the workflow will handle multiple files being uploaded simultaneously.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions