Maximum number of workers vs glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors


I have mentioned 4 as the Maximum number of workers when defining the Glue Spark Job of G1X worker type . If I check the Cloudwatch Job monitors , I see some these below metrics touching 20 + in the line graph .

  1. glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors
  2. glue.driver.ExecutorAllocationManager.executors.numberAllExecutors

My understanding is, for G1X worker type , there is only 1 executor per worker . Can someone explain if I am reading the chart right or if there is something missing ?

This is the line graph for 3 jobs , each having just 4 workers configured. Enter image description here

asked 3 months ago33 views
1 Answer
Accepted Answer

Please note that the metric- glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors does not depend upon the number of workers configured by you for the job. It is dynamically calculated as the job progresses. Generally this metric has a higher value at the beginning of the job. It just shows the maximum number of executors that are required in current moment to satisfy the workload. This is a spark related metric and it is calculated by making use of number of partitions, number of tasks per executor.

Please refer this documentation for more details regarding various metrics of a Glue job.

You can have a look at this documentation which shows how various calculations involving executors.

profile picture
answered 3 months ago
reviewed 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions