Error when Creating Iceberg Tables with Hidden Partitions using Athena: TOO_MANY_OPEN_WRITERS

Question

I have a set of Apache Iceberg tables that are logged in Glue Catalog. I was running some basic tests on Iceberg table partitions via Athena in us-east-1 without issue. We re-deployed the exact same pipeline, simply making the change to keep the iceberg buckets / glue data catalog within ca-central-1 instead of us-east-1. Now, when running the same partition tests in ca-central-1, I am getting hit with this error: ICEBERG_TOO_MANY_OPEN_PARTITIONS: Exceeded limit of 100 open writers for partitions.

The test is very simple:
I am just creating an empty copy of an existing table with hidden partitions defined. Then, I populate the partitioned table using the data from the original table. This way, I have a control to determine the impact of different partitioning strategies for various access patterns. These tests executed fine on the exact same data set in us-east-1, but are failing in ca-central-1 due to the issue shown above.

The configurations on the tables, s3 buckets, glue data catalog, and iceberg connection haven't changed (used the same cdk stack to deploy to different region). A little stuck on getting to the bottom of this issue. Is it possible that there are different Iceberg writer constraints per region?

Answer

Hello,

Athena has a limit of 100 partitions per CREATE TABLE AS SELECT (CTAS) query. Similarly, you can add a maximum of 100 partitions to a destination table with an INSERT INTO statement. This limit pertains only when the table is bucketed as well as partitioned. If you exceed this limitation, you may receive the error message HIVE_TOO_MANY_OPEN_PARTITIONS: Exceeded limit of 100 open writers for partitions/buckets.

You can refer below document for more details and how to mitigate this issue
[+] https://docs.aws.amazon.com/athena/latest/ug/ctas-insert-into.html

Error when Creating Iceberg Tables with Hidden Partitions using Athena: TOO_MANY_OPEN_WRITERS

Relevant content