Help with Cloud Shuffle Storage Plugin for EMR 7.9.0 / Spark 3.5.5

The Cloud Shuffle Storage Plugin for Apache Spark allows storing shuffle data on cloud storage systems like Amazon S3, which helps supplement or replace local disk storage capacity for large shuffle operations. This is particularly useful for avoiding "No space left on device" errors in large jobs.

For your migration to EMR 7.9.0 with Spark 3.5.5, you have a few options:

1. EMR Serverless with Shuffle-Optimized Disks: If you're considering EMR Serverless, you could use shuffle-optimized disks which offer higher IOPS and larger disk sizes. This is configured by setting parameters like spark.emr-serverless.executor.disk.type=shuffle_optimized in your Spark job configuration.

2. AWS Glue Spark Shuffle Manager: Another alternative is to use the AWS Glue Spark shuffle manager, which allows storing shuffle data in Amazon S3. This can be enabled with job parameters:

• --write-shuffle-files-to-s3 TRUE (main flag to use S3 for shuffle data)
• --write-shuffle-spills-to-s3 TRUE (optional, for offloading spill files to S3)
• --conf spark.shuffle.glue.s3ShuffleBucket=s3://<shuffle-bucket> (optional, to specify the S3 bucket)

3. Custom Implementation: Based on the compatibility issues noted with newer Spark versions, you might need to implement a custom version of the plugin that's compatible with Spark 3.5.5. This could involve modifying the source code to add any missing method implementations.

It's worth noting that there have been compatibility issues reported when using the Cloud Shuffle Storage Plugin with newer Spark versions. Specifically, there was an issue with Spark 3.4 where the plugin didn't implement a necessary method (commitAllPartitions), resulting in errors. Since you're moving to an even newer version (3.5.5), you might encounter similar compatibility challenges.

Before proceeding with a full migration, I'd recommend testing your approach with a small-scale deployment to ensure compatibility with your specific workload and Spark version.
Sources
Cloud Shuffle Storage Plugin for Apache Spark - AWS Glue
S3 shuffle storage for EMR Serverless | AWS re:Post
Building a reliable data pipeline - AWS Glue Best Practices: Building a Secure and Reliable Data Pipeline
Using shuffle-optimized disks - Amazon EMR