Does Kinesis Firehose destination support Glue table format of type Iceberg

0

Kinesis Firehose allows to configure S3 as a destination and in the Parquet section allows selecting a Glue catalog table of format Iceberg. However I had little to no luck querying the data.

Does Kinesis Firehose support Glue Iceberg as destination ? Any plans in near future if not.

RegeR
asked 10 days ago60 views
1 Answer
1
Accepted Answer

Currently, Kinesis Data Firehose does not natively support direct streaming to AWS Glue with Apache Iceberg table formats as a destination. Kinesis Data Firehose can output to Amazon S3 in formats such as Parquet, which can then be used in conjunction with AWS Glue for further processing and management in Iceberg format. Once the data is in S3, you can utilize AWS Glue to manage and catalog these data into Iceberg format, which supports complex and analytical workloads with robust data management features.

For your specific situation where you had difficulties querying data possibly streamed via Firehose and managed as Iceberg tables in AWS Glue, you might need to check the setup or integration process between S3, AWS Glue, and the Iceberg format. Ensure that the AWS Glue crawlers are properly configured to update the data catalog and that the Iceberg tables are correctly formatted and updated.

As of now, there are no official announcements from AWS about plans to directly integrate Kinesis Data Firehose with Iceberg formats in AWS Glue. You might want to follow AWS updates or explore using a custom solution involving AWS Lambda or another processing layer to handle data transformation before loading it into Iceberg tables via AWS Glue.

profile picture
EXPERT
answered 10 days ago
profile picture
EXPERT
reviewed 9 days ago
  • Thank you. It'd make sense to restrict the schema you can choose from Glue to the basic Hive based tables and omit the Iceberg tables during Firehose -> Parquet schema configuration to lessen the confusion with the customers.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions