Athena HIVE_METASTORE_ERROR when working with map<string, string> columns in parquet file


I have plenty of other Athena tables using parquet files, so the setup is okay, but this particular table has a column (metadata) that is of type map<string, string>. The parquet file - when loaded with pyarrow from the CLI correctly shows the table schema for that column:

study_id: string
subject_id: string
sample_id: string
collection_date: date32[day]
genotyping: string
metadata: map<string, string>
  child 0, entries: struct<key: string not null, value: string> not null
      child 0, key: string not null
      child 1, value: string

And I load load the parquet file just in back into Pandas as well. However, when I attempt to query the table with Athena I get this:

HIVE_METASTORE_ERROR: Error: type expected at the position 11 of 'map<string, string>' but ' ' is found.

I've been fighting this for several days now trying different things to no avail. How can I properly save this parquet file with the metadata column so that Athena can read it?

asked 10 months ago50 views