Issue with Apache Hudi and MoR snapshot tables

0

I just want to re-raise an issue we've been seeing for a long time, preventing us from querying Hudi MoR realtime tables using AWS Athena.

AWS Athena snapshot queries fails if there are two or more record array fields in a Hudi MoR table:

HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://prod-signal-data-platform-spark-experiments/temporary/articles_hudi_merge_on_read_dev/story_published_partition_date=2022-04-29/75f5af94-8369-4b04-b850-421826ef48cc-0_24-38-3759_20220506091903372.parquet (offset=33554432, length=33554432) using org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat: Can't redefine: array This query ran against the "articles" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 7a64950b-624d-4ea3-91d0-33be7a5e74b5

This corresponds to this Apache Hudi issue: https://github.com/apache/hudi/issues/3834 (fixed in Hudi 0.11.0) and requires an AWS Athena change.

I'd be great to see an AWS fix for this issue to allow us to fully leverage Hudi on AWS.

已提問 2 年前檢視次數 518 次
1 個回答
0

Thank you for bringing this up. Could you please raise a support ticket with Athena Premium Support so that we can help you create a feature request for this issue.

AWS
支援工程師
已回答 2 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南