Data in partitions on S3 bucket, getting below error in preview data query:

0

HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'site_rank' in table 'luci-data.luci-newslinkdata' is declared as type 'double', but partition 'partition_0=2011-12-30' declared column 'site_rank' as type 'bigint'. then, I followed the steps mentioned in: repost.aws/questions/QU1vPk1069Q5qg4iyuQWTk6Q/while-importing-s3-data-into-quicksight-from-glue-database-getting-the-following-error#ANMuJykKnKSD-jKoZk2tbvqw after updating in glue crawler: Update all new and existing partitions with metadata from the table Now the error is as follows: HIVE_BAD_DATA: Field site_rank's type INT64 in parquet file s3://rnvd-s3-luci-poc/NewsLinkData/2011-06-26/output_4404.parquet is incompatible with type double defined in table schema

asked 10 months ago256 views
1 Answer
0

What you are trying to do is not allowed, you cannot just change the types in the metadata if they don't match (or are compatible) with the actual data files.
If you don't want to update the data files, you can read as bigint and then cast in your query.
If you have mixture of parquet files with both types, the table is broken, replace/delete one of the set of files.

profile pictureAWS
EXPERT
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions