Step Function: Concurrent job is failing if any other job fails

Question

I am trying to trigger glue job via step functions by  passing parameters (there will be different table names) of  gule job through step function, so the issue is when one pram related glue job fails other concurrent job is stopping rather running independently

i have followed below approach to orchestrate my step function, can someone suggest

https://awstip.com/orchestrating-glue-jobs-using-aws-step-functions-to-perform-similar-etl-operations-on-multiple-d66275834162

Answer

Hello.

The behavior you're seeing (one failed job causing others to stop) typically occurs because of how your Step Functions state machine is defined. If a particular state in the state machine fails and there isn't appropriate error handling, it can cause the entire execution of the state machine to fail.

From the details you provided, it appears you are trying to run multiple Glue jobs concurrently. If you want each Glue job to run and complete independently of the others, you need to ensure your state machine is set up to handle errors appropriately.

Regards, Andrii

Step Function: Concurrent job is failing if any other job fails

Relevant content