Glue Iceberg Table optimisations fails

Question

I'm trying to use the automatic table optimisation feature for icerbg tables in AWS Glue. 
Sadly the tasks always fail:
Snapshot retention fails with: `SparkException occurred in retention optimizer: Cannot broadcast the table that is larger than 8GB: 8 GB. Root cause: Unknown`
Orphan file removal fails with: `SparkException occurred in orphan_file_deletion optimizer: Job aborted due to stage failure: Total size of serialized results of 117 tasks (1265.7 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB).  Root cause: Unknown`

My tables are quite big and filled for a long time, so I tried to aggresivly optimize them manually. I used this settings:
```
    'vacuum_min_snapshots_to_keep'='1',
    'vacuum_max_snapshot_age_seconds'='86400', 
    'vacuum_max_metadata_files_to_keep'='5'
```
But I still run into the same errors.

Is there a way to increase the thresholds which cause the tasks to fail. (I know how to do it for glue jobs, but for the automatic tasks, I did not find a setting for it). or is there another solution I could try?

re:Post Agent · Answer

The issues you're experiencing with AWS Glue's automatic table optimization for Iceberg tables are related to the size of your tables and the default limitations of the Spark engine used by AWS Glue. Here are some potential solutions and considerations:

1. For the snapshot retention failure:
The error suggests that the table metadata is too large to be broadcast (over 8GB). This is a limitation of Spark's broadcast join. Unfortunately, there isn't a direct way to increase this limit for the automatic optimization tasks.

2. For the orphan file deletion failure:
The error indicates that the result size exceeds the default Spark driver's maximum result size (1024 MB). While you mentioned you know how to adjust this for Glue jobs, the automatic optimization tasks don't provide a direct way to modify these Spark configurations.

Given these limitations, here are some approaches you can try:

1. Incremental optimization: Instead of trying to optimize the entire table at once, you could try to optimize it in smaller chunks. You might need to temporarily disable the automatic optimization and run manual optimization jobs that process subsets of your data.

2. Custom optimization job: Create a custom AWS Glue job that performs the optimization tasks. This would allow you to set Spark configurations like `spark.driver.maxResultSize` to accommodate your large tables.

3. Table partitioning: If your table isn't already partitioned, consider partitioning it. This can help reduce the amount of data processed in each optimization run.

4. More aggressive manual optimization: While you've already tried aggressive manual optimization, you might need to go even further. Consider reducing `vacuum_max_snapshot_age_seconds` even more if possible, or try running the vacuum operation multiple times to gradually reduce the table size.

5. Contact AWS Support: Given the limitations of the automatic optimization for very large tables, it might be worth contacting AWS Support to see if they have any additional recommendations or if there are any upcoming features to address these issues.

Remember, the automatic table optimization feature is designed to work with tables of varying sizes, but extremely large tables can still pose challenges. You may need to combine manual optimization techniques with the automatic feature to effectively manage your table sizes.

**Sources**  
[AWS Glue Data Catalog supports automatic optimization of Apache Iceberg tables through your Amazon VPC | AWS Big Data Blog](https://aws.amazon.com/blogs/big-data/aws-glue-data-catalog-supports-automatic-optimization-of-apache-iceberg-tables-through-your-amazon-vpc/)  
[Optimizing Iceberg tables - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/table-optimizers.html)  
[AWS Glue Data Catalog now supports storage optimization of Apache Iceberg tables - AWS](https://aws.amazon.com/about-aws/whats-new/2024/09/aws-glue-data-catalog-optimization-apache-iceberg-tables/)

Glue Iceberg Table optimisations fails

Add your answer

Relevant content