Skip to content

"Crawler does not support Iceberg tables" error

0

I have parquet files, catalogued in Glue as Iceberg DB. I can access them in Glue. But I cannot add them as a dataset in DataBrew. It says they aren't crawled. When I try to crawl them, I get a "Crawler does not support Iceberg tables" error. When I google the error, all I find are articles about how crawler now works with Iceberg.

asked a year ago270 views

1 Answer
0

The issue you're experiencing is related to how AWS services interact with Apache Iceberg tables. While AWS Glue crawlers do support Iceberg tables (which is why you see articles mentioning this capability), there appears to be a compatibility issue between AWS DataBrew and Iceberg-formatted tables.

Here's what's happening:

  1. You have successfully cataloged your parquet files as Iceberg tables in the AWS Glue Data Catalog.

  2. When trying to add these tables as a dataset in DataBrew, you're encountering an error suggesting the tables aren't properly crawled.

  3. When attempting to re-crawl them, you're receiving a "Crawler does not support Iceberg tables" error, which contradicts the documentation stating that Glue crawlers support Iceberg.

This discrepancy suggests that while AWS Glue itself supports Iceberg tables, AWS DataBrew may not yet fully support Iceberg as a source format. DataBrew is a relatively newer service compared to Glue, and it might not have complete integration with all the table formats that Glue supports.

To work around this issue, you could try:

  1. Creating a standard (non-Iceberg) Glue table that points to the same underlying parquet files. This would involve creating a new table definition in Glue that uses the parquet format directly rather than Iceberg.

  2. Using AWS Glue ETL jobs to convert your Iceberg tables to a format that DataBrew fully supports.

  3. Using Athena or other AWS analytics services that have better support for Iceberg tables if your workflow allows.

If you need to specifically use DataBrew with these datasets, you might need to contact AWS Support for further assistance, as this appears to be a service limitation rather than a configuration issue on your end.
Sources
Unable to view Iceberg table data in Athena despite successful Glue crawl | AWS re:Post
Does Kinesis Firehose destination support Glue table format of type Iceberg | AWS re:Post

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.