The parquet has a nested structure where a map contains a struct which is itself nested.
i'm not sure if athen'as serDe can handle it, but i would like to find a way to read it with athena.
The table creation worked fine, and the structure i described matches the parquet.
CREATE EXTERNAL TABLE amitworkloads14(
name
string,
....
containers
map<string,struct<
name:string,
......
stats:struct<
lastCpuReq:double,
....
oomEventsTimed:map<string,double>,
cpuThrottleTimed:map<string,struct<
avgThrottle:double,
...
>>,
...
>
,
maxInstances
bigint,
...
runningMinutesByNodeGroup
map<string,bigint>
)
STORED AS PARQUET
LOCATION 's3://dev-customers-data/test/new-pqt-lib/workloads/'
tblproperties ("parquet.compression"="SNAPPY")
The full error message:
HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://dev-customers-data/test/new-pqt-lib/workloads/new_workloads.parquet (offset=0, length=2227521): required group stats { ..................... } is not primitive
This query ran against the "new-pqt-lib" database, unless qualified by the query. Please post the error message on our forum
or contact customer support
with Query Id: 4a72c1ed-fd99-4149-9f57-af240ccbef6b