Glue run concurrent jobs.

Question

We have a job (Jupyter notebook job) version 4 that we are trying to run in concurrent mode changing some of the parameters and running via AWS CLI 
like below

```
aws glue start-job-run --job-name "country-job" --arguments='--ISO=DNK, --DAYS=2' --profile glue-eu
aws glue start-job-run --job-name "country-job" --arguments='--ISO=NOR, --DAYS=2' --profile glue-eu
```

Added the below to the job %%configure including the magic "max_concurrent_runs": 4

```
%%configure
{
    "region": "eu-west-1",
    "idle_timeout": "480",
    "glue_version": "4.0",
    "number_of_workers": 10,
    "execution_class": "FLEX",
    "iam_role": "arn:aws:iam::ABC:role/AWSGlueServiceRole-searchlogs-s3",
    # "profile": "glue-profile",
    "session_id_prefix": "ABC",
    "worker_type": "G.1X",
    "max_concurrent_runs": 4
}
```

But keeps failing with the error below 
`An error occurred (ConcurrentRunsExceededException) when calling the StartJobRun operation: Concurrent runs exceeded for country-job`

Do I need to set the job any diferently ? 
Thanks

Answer

Your Magic looks good and it will allow you to run 4 instance of the job concurrently.

1. Check the concurrency is configured from job property with below command, and the output with 4 MaxConcurrentRuns : 
aws glue get-job --job-name "country-job" --profile glue-eu
```
"ExecutionProperty": {
            "MaxConcurrentRuns": 4
        }
```
2. Check the run instance not crossing 4 from "Runs" or Glue Monitoring console. 
3. Finally, remember after the Glue job completes (SUCCEEDED or FAILED), it needs few seconds (around 5-10 seconds) to completely cleanup resources, until then it considered running. This is because the job metering stopped for customer, but it can still conflict with concurrency. So either slow down on the request for new run or have little higher concurrency count.

Glue run concurrent jobs.

Relevant content