Glue: No autoscaling while option enabled
We are running glue 3.0 jobs with the --enable-auto-scaling set but according to the metrics, the number of active executors is not reduced while it should be according to the number of maximum needed executor. It is working sometimes but clearly not as we would expect.
We're seeing that on lots of jobs causing major cost impact.
Has anyone experienced such kind of issues or could help us understand what could cause this?
Here is an example to illustrate our issue:
Thanks, Alain
- Newest
- Most votes
- Most comments
My contact at AWS suggested to open a ticket and post the issue in re:Post. Case ID: 10433082981 I just shared the spark UI logs in the ticket. I can share the log on demand in private messages. Alain
Hello Alain,
I understand that you have enabled auto scaling for the Glue job, but from the CloudWatch metrics it seems like it has not scaled down the executors throughout the job. To further investigate this issue, we would require the Spark History Server logs and the job run. For more information, see Enabling the Apache Spark web UI for AWS Glue jobs.
Please, feel free to open a support case with AWS using the following link with the Spark History Server logs, the job run and we would be happy to help.
Relevant content
- Accepted Answerasked a year ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated a year ago
Hello Alain,
I checked the case ID shared by you and can see that you have shared the logs. We will get back to you on the same case ID. If you have any additional queries regarding this please feel free to get back to me. Thanks for your patience and understanding on the case.