Is Glue Spark shuffle manager supported with Glue 3.0?

Question

Is Glue Spark shuffle manager supported with Glue 3.0? I'm trying to use `--write-shuffle-files-to-s3` flag with a Glue 3.0 job and it doesn't seem to work as no files are written.

If it's not supported - will it be supported eventually?

Accepted Answer

AWS Glue Spark Shuffle Manager is currently only supported on AWS Glue version 2.0, and Spark 2.4.3. as mentioned in documentation link [1] under "Notes and limitations" Section. Unfortunately this feature will not work if your Glue job is using Glue version 3.0.

I understand that after downgrading the Glue job to Glue version 2.0 you were able to make the shuffle work.

Regarding your query, whether it be supported by Glue 3.0, please note that it is a pending feature request with the Glue team. Unfortunately, there is no ETA or time frame on when this feature or if these features will be rolled out by our service team as each release cycle of features is decided by the demand and urgency of the features. Although, we do appreciate your feedback.

You can keep an eye on our "What's New with AWS" blog to monitor our new releases as they become available.

[+] https://aws.amazon.com/new/

———

Reference:

[1] AWS Glue Spark shuffle manager with Amazon S3 - https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shuffle-manager.html

Is Glue Spark shuffle manager supported with Glue 3.0?

相关内容