Glue Python job runs multiple times with different arguments

0

hi Team ,

My requirement is , let's say I have 10 different glue pyspark jobs ( job1, job2,.............job10) , once job1 execute and succeed then it start next glue python shell ( let's say name of script is : glue_common_python_shell.py ) Glue python shell will be one script with different arguments each time.

in other words , glue pyspark job1 completes -- >trigger --> glue_common_python_shell.py ( with arguments --table_name1) glue pyspark job2 completes -- >trigger --> glue_common_python_shell.py ( with arguments --table_name2) ...................... glue pyspark job10 completes -- >trigger --> glue_common_python_shell.py ( with arguments --table_name10)

how can I achieve this orchestration in AWS .Please help.

Thanks in advance.

asked 2 years ago2693 views
1 Answer
1
Accepted Answer

There are multiple ways to orchestrate the Glue jobs. I will list an example architectures that would work in this scenario:

Use StepFunctions to execute the series of Glue jobs. In the below example, the Lambda function returns all table names and other inputs needed for the Glue job and those can be passed into Glue jobs as noted below. The startJobRun.sync ensures that the next steps starts only after the job is successful and complete.

{
  "Comment": "Run Glue job workflow",
  "StartAt": "Lambda Invoke",
  "States": {
    "Lambda Invoke": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "OutputPath": "$.Payload",
      "Parameters": {
        "Payload.$": "$"
      },
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Next": "run_glue_job_1"
    },
    "run_glue_job_1": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "glue_job_1"
      },
      "Next": "run_glue_job_2",
      "InputPath": "$.table_name_1"
    },
    "run_glue_job_2": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "glue_job_2"
      },
      "Next": "run_glue_job_3",
      "InputPath": "$.table_name_2"
    },
    "run_glue_job_3": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "glue_job_3"
      },
      "End": true,
      "InputPath": "$.table_name_3"
    }
  }
}

Glue workflow does not have the functionality yet to run the same job multiple times with different parameters. You could create multiple workflows and each workflow would call the same job with different parameters and would be triggered based on previous workflow completion.

You could use Lambda's to trigger the job using input parameters and wait for a few minutes and check for job completion then trigger the next job. The problem with this approach is that Lambda would use up compute resources when waiting for the glue job to complete.

Some other ideas may include using Airflow(MWAA) to orchestrate.

profile pictureAWS
answered 2 years ago
AWS
EXPERT
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions