- Newest
- Most votes
- Most comments
AWS Glue Spark Shuffle Manager is currently only supported on AWS Glue version 2.0, and Spark 2.4.3. as mentioned in documentation link [1] under "Notes and limitations" Section. Unfortunately this feature will not work if your Glue job is using Glue version 3.0.
I understand that after downgrading the Glue job to Glue version 2.0 you were able to make the shuffle work.
Regarding your query, whether it be supported by Glue 3.0, please note that it is a pending feature request with the Glue team. Unfortunately, there is no ETA or time frame on when this feature or if these features will be rolled out by our service team as each release cycle of features is decided by the demand and urgency of the features. Although, we do appreciate your feedback.
You can keep an eye on our "What's New with AWS" blog to monitor our new releases as they become available.
[+] https://aws.amazon.com/new/
———
Reference:
[1] AWS Glue Spark shuffle manager with Amazon S3 - https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shuffle-manager.html
Relevant content
- asked 2 years ago
- asked 9 months ago
- Accepted Answerasked 5 months ago
- asked 2 years ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated a year ago
Update: after changing the job to Glue 2.0 shuffle seems to work (at least shuffle files are written). So the question is now whether this will be supported in Glue 3.0.