CTAS can't handle empty arrays or maps

When using the new CTAS functionality to convert JSON to PARQUET, I kept getting this error:

GENERIC_INTERNAL_ERROR: Parquet record is malformed: empty fields are illegal, the field should be ommited completely instead. You may need to manually clean the data at location '{scrubbed}' before retrying. Athena will not delete data in your account.

This query ran against the "sampledb" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: bfe7eeaf-fe0a-4936-8515-8a9c2d1b22d0.

I eventually realized this always happens if any of the JSON is an empty array "[]" or there is a map type that is an empty map "{}". Is this a know limitation? It would be great if this were a little more flexible/robust.

A sample table:

CREATE EXTERNAL TABLE sampledb.test_empty_object (
  `name` string,
  `fields` map <string, string>,
  `friends` array <string>,
  `data`struct <
     name: string,
     value: string
  >
)
ROW FORMAT  serde 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ('ignore.malformed.json' = 'true')
LOCATION 's3://somewhere'

sample CTAS query

CREATE TABLE sampledb.test_empty_array_parquet
WITH (
     format = 'PARQUET', 
     external_location = 's3://somewhere'
)
AS SELECT *
FROM sampledb.test_empty_array

It handles "data" being {} just fine, but if fields = {} or friends = [], the error above is thrown.

Edited by: JeffreyBarrus on Oct 25, 2018 9:26 AM