I have a set of Apache Iceberg tables that are logged in Glue Catalog. I was running some basic tests on Iceberg table partitions via Athena in us-east-1 without issue. We re-deployed the exact same pipeline, simply making the change to keep the iceberg buckets / glue data catalog within ca-central-1 instead of us-east-1. Now, when running the same partition tests in ca-central-1, I am getting hit with this error: ICEBERG_TOO_MANY_OPEN_PARTITIONS: Exceeded limit of 100 open writers for partitions.
The test is very simple:
I am just creating an empty copy of an existing table with hidden partitions defined. Then, I populate the partitioned table using the data from the original table. This way, I have a control to determine the impact of different partitioning strategies for various access patterns. These tests executed fine on the exact same data set in us-east-1, but are failing in ca-central-1 due to the issue shown above.
The configurations on the tables, s3 buckets, glue data catalog, and iceberg connection haven't changed (used the same cdk stack to deploy to different region). A little stuck on getting to the bottom of this issue. Is it possible that there are different Iceberg writer constraints per region?
