By using AWS re:Post, you agree to the Terms of Use

Is Glue Spark shuffle manager supported with Glue 3.0?

0

Is Glue Spark shuffle manager supported with Glue 3.0? I'm trying to use --write-shuffle-files-to-s3 flag with a Glue 3.0 job and it doesn't seem to work as no files are written.

If it's not supported - will it be supported eventually?

  • Update: after changing the job to Glue 2.0 shuffle seems to work (at least shuffle files are written). So the question is now whether this will be supported in Glue 3.0.

1 Answer
0
Accepted Answer

AWS Glue Spark Shuffle Manager is currently only supported on AWS Glue version 2.0, and Spark 2.4.3. as mentioned in documentation link [1] under "Notes and limitations" Section. Unfortunately this feature will not work if your Glue job is using Glue version 3.0.

I understand that after downgrading the Glue job to Glue version 2.0 you were able to make the shuffle work.

Regarding your query, whether it be supported by Glue 3.0, please note that it is a pending feature request with the Glue team. Unfortunately, there is no ETA or time frame on when this feature or if these features will be rolled out by our service team as each release cycle of features is decided by the demand and urgency of the features. Although, we do appreciate your feedback.

You can keep an eye on our "What's New with AWS" blog to monitor our new releases as they become available.

[+] https://aws.amazon.com/new/

———

Reference:

[1] AWS Glue Spark shuffle manager with Amazon S3 - https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shuffle-manager.html

SUPPORT ENGINEER
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions