Athena HIVE_METASTORE_ERROR when working with map<string, string> columns in parquet file

0

I have plenty of other Athena tables using parquet files, so the setup is okay, but this particular table has a column (metadata) that is of type map<string, string>. The parquet file - when loaded with pyarrow from the CLI correctly shows the table schema for that column:

study_id: string
subject_id: string
sample_id: string
collection_date: date32[day]
genotyping: string
metadata: map<string, string>
  child 0, entries: struct<key: string not null, value: string> not null
      child 0, key: string not null
      child 1, value: string

And I load load the parquet file just in back into Pandas as well. However, when I attempt to query the table with Athena I get this:

HIVE_METASTORE_ERROR: Error: type expected at the position 11 of 'map<string, string>' but ' ' is found.

I've been fighting this for several days now trying different things to no avail. How can I properly save this parquet file with the metadata column so that Athena can read it?

gefragt vor 2 Jahren154 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen