HIVE_PARTITION_SCHEMA_MISMATCH - Athena error on S3 Parquet file
Getting below error while querying S3 parquet file from Athena. The schema has changed from timestamp to string for one of the columns, is there any workaround to have the latest schema changed to string (Alter table and msck repair table didn't work).
HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'birth_year' in table 'curated_ndb_datapipeline.registered_nurse' is declared as type 'timestamp', but partition 'data_year=2021/prov_terr_reg=959' declared column 'birth_year' as type 'string'
If all partitions have the same schema with the column defined as string and only the table has the column defined as timestamp you could try to:
- run a Glue crawler and see if it correctly updates the table
- use the Glue APis to update the column data type where needed (if it is only at table level you could consider to do it in the console).
hope this helps.
Relevant questions
HIVE_PARTITION_SCHEMA_MISMATCH - Athena error on S3 Parquet file
asked a month agoHIVE_BAD_DATA with Parquet BINARY
asked 5 years agoAthena : HIVE_BAD_DATA
asked 4 months agoAthena HIVE_METASTORE_ERROR when working with map<string, string> columns in parquet file
asked 3 months agoQueries fail when schema has array type
asked 5 years agoNot able to read S3 Parquet file
asked a month agoHIVE_UNKNOWN_ERROR: Duplicate key string
asked 3 months agoHIVE_FILESYSTEM_ERROR: Incorrect fileSize
asked 3 months agoBest way to overcome HIVE_PARTITION_SCHEMA_MISMATCH error in Athena while preserving structure of structs?
asked a month agoDMS MSSQL Server to S3 in Parquet format datetime to timestamp issue
asked 3 years ago