Glue queue: Max concurrent runs exceeded

Hi,

You should have a look at the Service Quotas. On the left panel, click on AWS services and search for Glue. I’m not entirely sure which quota name you’ll need to increase, but it might be:

• Max concurrent job runs per account

Hope this helps 🙂

I understand your frustration with the AWS Glue job queuing feature not working as expected. Let me clarify how the job queuing feature works and why you might still be seeing the "Max concurrent runs exceeded" error.

AWS Glue job queuing is designed to help manage concurrent job runs within your account's service quotas and limits. When enabled, it should automatically queue job runs that exceed these limits, putting them in a "WAITING" state until resources become available.

However, there are a few important points to consider:

1. Job queuing applies to certain types of limits, including max concurrent job runs per account, max concurrent Data Processing Units (DPUs), and resource unavailability in Amazon VPC.

2. There's also a limit on max concurrent job runs per job, which is controlled by the MaxConcurrentRuns parameter for each job. This limit is separate from the account-level quota.

3. The queuing mechanism has its own limits. Jobs may queue for up to 15 minutes or make 10 retry attempts, whichever comes first.

Given that you're seeing some jobs go to "WAITING" status while others fail, it's likely that you're hitting multiple limits simultaneously. The jobs that go to "WAITING" are probably queued due to account-level quotas, while the ones that fail might be exceeding the per-job concurrent run limit.

To address this issue:

1. Check your MaxConcurrentRuns setting for each job. This parameter sets the maximum number of concurrent runs allowed for that specific job.

2. Review your account-level quotas for max concurrent job runs and max DPUs.

3. Consider implementing a backoff strategy in your job submission process to avoid overwhelming the system.

4. Monitor your job runs closely to identify patterns in which jobs are queuing versus failing.

5. If you consistently need more concurrent runs, you may want to request a quota increase from AWS support.

Remember, the job queuing feature is designed to help manage concurrency, but it doesn't eliminate all concurrency-related errors, especially when dealing with job-specific limits or when the queuing capacity itself is exceeded. By fine-tuning your job configurations and understanding the various limits at play, you should be able to better leverage the job queuing feature and reduce the occurrence of concurrency-related failures.
Sources
AWS Glue job run statuses - AWS Glue
AWS Glue now provides job queuing - AWS
Introducing job queuing to scale your AWS Glue workloads | AWS Big Data Blog
Jobs - AWS Glue