- Newest
- Most votes
- Most comments
Even if you have a dynamic schema, your ETL job has to make sure that when you insert data into the table, the schema is compatible, for instance converting that array of string to an array of structs (even if you have a generic property name).
BTW, you shouldn't need a crawler for that. The ETL job could read the json directly and write directly into the table, that way it's likely you would detect these issues at ingestion.
This is my issue, my Array column in Glue table might be empty in one partition and might have a value.
HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'edx' in table 'datalake.test' is declared as type 'array<array<string>>', but partition 'partition_0=02' declared column 'col0' as type 'array<array<struct<cdd:array<structcddc:string,cdds:string>,cdir:string,dvr:string,erce:string,epr:string,edir:string,eds:string,edts:string,edd:string,mts:string,mud:string>>>'.
Yeah, that's not a valid table, in the code make sure the output schema is compatible, transforming it if needed
Relevant content
- asked 8 months ago
- asked 8 months ago
- asked 3 months ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
I have a deeply nested JSON, and I'm extracting the columns that I needed and converted them to a parquet file. I'm using the crawler for incremental ingestion of data and effective querying through Athena. Are you saying there is a way around ?
If you are going to read the JSON to convert it (not to run SQL queries or similar), I wouldn't bother putting a table on top, I would read from S3 directly and adapt the inferred schema on the fly