By using AWS re:Post, you agree to the Terms of Use

Glue: No autoscaling while option enabled

0

We are running glue 3.0 jobs with the --enable-auto-scaling set but according to the metrics, the number of active executors is not reduced while it should be according to the number of maximum needed executor. It is working sometimes but clearly not as we would expect.

We're seeing that on lots of jobs causing major cost impact.

Has anyone experienced such kind of issues or could help us understand what could cause this?

Here is an example to illustrate our issue: Enter image description here

Thanks, Alain

2 Answers
0

My contact at AWS suggested to open a ticket and post the issue in re:Post. Case ID: 10433082981 I just shared the spark UI logs in the ticket. I can share the log on demand in private messages. Alain

answered 23 days ago
  • Hello Alain,

    I checked the case ID shared by you and can see that you have shared the logs. We will get back to you on the same case ID. If you have any additional queries regarding this please feel free to get back to me. Thanks for your patience and understanding on the case.

0

Hello Alain,

I understand that you have enabled auto scaling for the Glue job, but from the CloudWatch metrics it seems like it has not scaled down the executors throughout the job. To further investigate this issue, we would require the Spark History Server logs and the job run. For more information, see Enabling the Apache Spark web UI for AWS Glue jobs.

Please, feel free to open a support case with AWS using the following link with the Spark History Server logs, the job run and we would be happy to help.

SUPPORT ENGINEER
answered 23 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions