HIVE_PARTITION_SCHEMA_MISMATCH - Athena error on S3 Parquet file

Question

Getting below error while querying S3 parquet file from Athena. The schema has changed from timestamp to string for one of the columns, is there any workaround to have the latest schema changed to string (Alter table and msck repair table didn't work).

HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'birth_year' in table 'curated_ndb_datapipeline.registered_nurse' is declared as type 'timestamp', but partition 'data_year=2021/prov_terr_reg=959' declared column 'birth_year' as type 'string'

Answer

If all partitions have the same schema with the column defined as string and only the table has the column defined as timestamp you could try to:

1. run a Glue crawler and see if it correctly updates the table
2. use the Glue APis to update the column data type where needed (if it is only at table level you could consider to do it in the console).

hope this helps.

HIVE_PARTITION_SCHEMA_MISMATCH - Athena error on S3 Parquet file

Relevanter Inhalt