Please note that the metric- glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors does not depend upon the number of workers configured by you for the job. It is dynamically calculated as the job progresses. Generally this metric has a higher value at the beginning of the job. It just shows the maximum number of executors that are required in current moment to satisfy the workload. This is a spark related metric and it is calculated by making use of number of partitions, number of tasks per executor.
Please refer this documentation for more details regarding various metrics of a Glue job.
You can have a look at this documentation which shows how various calculations involving executors.
Glue job does not sort data without "Automatically scale the number of workers" checkAccepted Answerasked 24 days ago
Paying a bonus to a large number of workersasked 10 months ago
Maximum number of Jobs queuedasked 3 years ago
Maximum number of workers vs glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutorsAccepted Answerasked 3 months ago
Glue job fail many workersasked 8 months ago
Get total number of array workers inside of array jobasked a year ago
How does Glue divide work among workersasked 6 months ago
AWS Glue - Glue Jobs - Glue 2.0: Worker Typesasked 9 months ago
Maximum number of databases?asked 4 years ago
Serverless Inference - Limit number of workersasked 10 months ago