Athena error HIVE_BAD_DATA: Not valid Parquet file

0

Hello,

I created a glue table with parquet serde initially.

Details:
input format: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
output format: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
serde: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
classification: parquet

Then I edited the table manually via console to use json serde.

Details:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.openx.data.jsonserde.JsonSerDe
classification: json

The underlying s3 data is in json format.

Now when I try to query the data via Athena, I get the below error.

HIVE_BAD_DATA: Not valid Parquet file

It seems Athena is not picking up the updated glue details? I tried MSCK repair on the table but it does not work.

Is there a way I can fix it without deleting the table?

Thanks.

質問済み 1年前1473ビュー
1回答
0

Figured it.

If I drop the existing partitions and add it, Athena does not give error anymore.

I also tried to add more partitions after this, new partitions are taking json serde now.

回答済み 1年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン