Glue Python job runs multiple times with different arguments

There are multiple ways to orchestrate the Glue jobs. I will list an example architectures that would work in this scenario:

Use StepFunctions to execute the series of Glue jobs. In the below example, the Lambda function returns all table names and other inputs needed for the Glue job and those can be passed into Glue jobs as noted below. The startJobRun.sync ensures that the next steps starts only after the job is successful and complete.

{
  "Comment": "Run Glue job workflow",
  "StartAt": "Lambda Invoke",
  "States": {
    "Lambda Invoke": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "OutputPath": "$.Payload",
      "Parameters": {
        "Payload.$": "$"
      },
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Next": "run_glue_job_1"
    },
    "run_glue_job_1": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "glue_job_1"
      },
      "Next": "run_glue_job_2",
      "InputPath": "$.table_name_1"
    },
    "run_glue_job_2": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "glue_job_2"
      },
      "Next": "run_glue_job_3",
      "InputPath": "$.table_name_2"
    },
    "run_glue_job_3": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "glue_job_3"
      },
      "End": true,
      "InputPath": "$.table_name_3"
    }
  }
}

Glue workflow does not have the functionality yet to run the same job multiple times with different parameters. You could create multiple workflows and each workflow would call the same job with different parameters and would be triggered based on previous workflow completion.

You could use Lambda's to trigger the job using input parameters and wait for a few minutes and check for job completion then trigger the next job. The problem with this approach is that Lambda would use up compute resources when waiting for the glue job to complete.

Some other ideas may include using Airflow(MWAA) to orchestrate.