Maximum number of workers vs glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors
I have mentioned 4 as the Maximum number of workers when defining the Glue Spark Job of G1X worker type . If I check the Cloudwatch Job monitors , I see some these below metrics touching 20 + in the line graph .
- glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors
- glue.driver.ExecutorAllocationManager.executors.numberAllExecutors
My understanding is, for G1X worker type , there is only 1 executor per worker . Can someone explain if I am reading the chart right or if there is something missing ?
This is the line graph for 3 jobs , each having just 4 workers configured.
- 新しい順
- 投票が多い順
- コメントが多い順
Please note that the metric- glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors does not depend upon the number of workers configured by you for the job. It is dynamically calculated as the job progresses. Generally this metric has a higher value at the beginning of the job. It just shows the maximum number of executors that are required in current moment to satisfy the workload. This is a spark related metric and it is calculated by making use of number of partitions, number of tasks per executor.
Please refer this documentation for more details regarding various metrics of a Glue job.
You can have a look at this documentation which shows how various calculations involving executors.
