HIVE_PARTITION_SCHEMA_MISMATCH

0

I'm converting my json files with dynamic schema to Parquet and used a Glue Crawler to create Glue Table to query using Athena. I got the below error as one column in my table got array<string> in one partition and array<struct> in another.

HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced.

How to solve the issue?

Magneto
已提问 4 个月前379 查看次数
2 回答
0

Even if you have a dynamic schema, your ETL job has to make sure that when you insert data into the table, the schema is compatible, for instance converting that array of string to an array of structs (even if you have a generic property name).
BTW, you shouldn't need a crawler for that. The ETL job could read the json directly and write directly into the table, that way it's likely you would detect these issues at ingestion.

profile pictureAWS
专家
已回答 4 个月前
  • I have a deeply nested JSON, and I'm extracting the columns that I needed and converted them to a parquet file. I'm using the crawler for incremental ingestion of data and effective querying through Athena. Are you saying there is a way around ?

  • If you are going to read the JSON to convert it (not to run SQL queries or similar), I wouldn't bother putting a table on top, I would read from S3 directly and adapt the inferred schema on the fly

0

This is my issue, my Array column in Glue table might be empty in one partition and might have a value.

HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'edx' in table 'datalake.test' is declared as type 'array<array<string>>', but partition 'partition_0=02' declared column 'col0' as type 'array<array<struct<cdd:array<structcddc:string,cdds:string>,cdir:string,dvr:string,erce:string,epr:string,edir:string,eds:string,edts:string,edd:string,mts:string,mud:string>>>'.

Magneto
已回答 4 个月前
  • Yeah, that's not a valid table, in the code make sure the output schema is compatible, transforming it if needed

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则