I have plenty of other Athena tables using parquet files, so the setup is okay, but this particular table has a column (metadata) that is of type map<string, string>
. The parquet file - when loaded with pyarrow from the CLI correctly shows the table schema for that column:
study_id: string
subject_id: string
sample_id: string
collection_date: date32[day]
genotyping: string
metadata: map<string, string>
child 0, entries: struct<key: string not null, value: string> not null
child 0, key: string not null
child 1, value: string
And I load load the parquet file just in back into Pandas as well. However, when I attempt to query the table with Athena I get this:
HIVE_METASTORE_ERROR: Error: type expected at the position 11 of 'map<string, string>' but ' ' is found.
I've been fighting this for several days now trying different things to no avail. How can I properly save this parquet file with the metadata column so that Athena can read it?