Partition size in Glue

Your understanding of the AWS Glue worker configuration is partially correct, but there are some important considerations to keep in mind when determining optimal partition sizes.

While it's true that G.1X workers in AWS Glue have 4 vCPU cores and approximately 10GB of heap storage, the optimal partition size is not necessarily 750MB as you suggested. The recommended partition size for AWS Glue jobs is typically between 128MB to 1GB, with 128MB being a good starting point for most scenarios.

Here's why:

1. Spark's parallel processing: Spark, which AWS Glue uses, is designed to process data in parallel across multiple cores and workers. Smaller partitions (like 128MB) allow for better distribution of work across available resources.

2. I/O efficiency: While larger partitions might seem more efficient for processing, they can lead to increased I/O time when reading from storage. Smaller partitions allow for more efficient data retrieval and processing.

3. Memory management: Although each core has access to a portion of the execution memory, Spark manages memory dynamically. Larger partitions could potentially cause memory pressure and lead to spilling to disk, which degrades performance.

4. Flexibility: Smaller partitions provide more flexibility for Spark to optimize task distribution and parallelism across available resources.

5. Data skew: Larger partitions may exacerbate issues with data skew, where some partitions contain significantly more data than others, leading to uneven processing times.

That being said, the optimal partition size can vary depending on your specific workload, data characteristics, and the nature of your transformations. While 128MB is a good starting point, you may need to experiment with different partition sizes to find the optimal balance for your particular use case.

For operations like group by or joins, it's generally more beneficial to focus on even data distribution rather than maximizing partition size. Ensuring that your data is well-partitioned based on the join or group by keys can often yield better performance improvements than increasing partition size.

Remember, AWS Glue is designed to handle partitioning automatically in many cases. If you find that you need to manually adjust partitions, it's best to start with the recommended range (128MB to 1GB) and fine-tune based on performance metrics and your specific workload characteristics.
Sources
Building a performance efficient data pipeline - AWS Glue Best Practices: Building a Performant and Cost Optimized Data Pipeline
Parallelize tasks -
AWS Glue ETL - AWS Prescriptive Guidance