Athena / Glue pySpark: Drop Iceberg Partition Field not possible

You are correct that dropping partition fields from Iceberg tables is not currently supported in Amazon Athena or through Spark SQL queries in AWS Glue.

Athena has limited support for Iceberg table operations, and dropping partition fields is not among the supported operations. The ALTER TABLE DROP PARTITION FIELD command is not available in Athena's implementation of Iceberg support.

Similarly, when using Spark SQL through AWS Glue or EMR, you may encounter limitations when trying to modify the partition structure of existing Iceberg tables.

If you need to modify the partition structure of an Iceberg table, you might need to consider alternative approaches:

1. Create a new table with the desired partition structure and copy the data over.
2. Use Apache Spark directly (not through Athena or Glue) if you have access to an environment that supports the latest Iceberg features.
3. Use AWS Glue ETL jobs to rewrite the table with a new partition structure.

It's worth noting that modifying partition fields, especially removing them, can be a complex operation that may require rewriting the entire dataset. This is why such operations are often limited or unsupported in managed services like Athena.

If this functionality is critical for your use case, you may want to provide feedback to the AWS Athena team about your need for more comprehensive Iceberg partition evolution support.
Sources
Athena Iceberg partitioning on nested ROW fields | AWS re:Post
How to add partition fields to Iceberg table | AWS re:Post
Unsupported DDL - Amazon Athena