- 1 Standard node = 1 DPU. ( https://docs.aws.amazon.com/glue/latest/dg/add-job.html )
- on one standard node you have 2 executor , each executor gets assigned half the cores and half the memory (actually less you have to consider the overhead)
- has mentioned in answer 2 - 2 executors will need to split cores and memory , so you run more tasks in parallel but each can only manage a smaller amount of data (remember that Spark tries to run all in memory) . when you have a single executor you reduce the parallel tasks but increase the data each task can manage. Starting from Glue 2.0 we advice to use only G.1X and G.2X worker types.
- Glue 2.0 does not use dynamic executor allocation, so you need to specify the exact number of Worker. Glue 1.0 was using Dynamic executor allocation so in theory the job would define how many executor were needed , so theoretically you are right., but unfortunately, I am not sure if that matched the billing process. (anyway, the current advice it to use Glue 2.0 and Glue 3.0 , so I would not focus on Glue 1.0 configuration).
With Glue 3.0 we have launched in preview autoscaling that will allow you to define again a maximum number of workers but use actual workload statistics to define the number of workers active (AND billed) at anytime.
hope this helps
Will AWS Glue Blueprints support trigger types other than a schedule?Accepted Answerasked 7 months ago
Can Glue bookmarks be used between Glue workflow steps?Accepted Answerasked 3 years ago
How to launch Glue Jobs from CloudFormation?asked 2 years ago
"Resource Unavailable" in AWS GlueAccepted AnswerEXPERTasked 2 years ago
AWS Glue is taking very long time to read data from MySQL table (61 millions)asked 2 months ago
Runaway glue jobs leading to exception Task allocated capacity limit being exceeded ? "Glue ErrorCode 400 InvalidInputException"Accepted Answerasked 9 days ago
Sophisticated Triggering of Glue Jobsasked 5 months ago
Creating AWS Glue JObs using eclipse IDE Javaasked a month ago
AWS Glue - Glue Jobs - Glue 2.0: Worker Typesasked 7 months ago
Glue Jobs & Multiple tablesAccepted Answerasked 4 years ago