- Newest
- Most votes
- Most comments
The issue you're experiencing is related to how AWS services interact with Apache Iceberg tables. While AWS Glue crawlers do support Iceberg tables (which is why you see articles mentioning this capability), there appears to be a compatibility issue between AWS DataBrew and Iceberg-formatted tables.
Here's what's happening:
-
You have successfully cataloged your parquet files as Iceberg tables in the AWS Glue Data Catalog.
-
When trying to add these tables as a dataset in DataBrew, you're encountering an error suggesting the tables aren't properly crawled.
-
When attempting to re-crawl them, you're receiving a "Crawler does not support Iceberg tables" error, which contradicts the documentation stating that Glue crawlers support Iceberg.
This discrepancy suggests that while AWS Glue itself supports Iceberg tables, AWS DataBrew may not yet fully support Iceberg as a source format. DataBrew is a relatively newer service compared to Glue, and it might not have complete integration with all the table formats that Glue supports.
To work around this issue, you could try:
-
Creating a standard (non-Iceberg) Glue table that points to the same underlying parquet files. This would involve creating a new table definition in Glue that uses the parquet format directly rather than Iceberg.
-
Using AWS Glue ETL jobs to convert your Iceberg tables to a format that DataBrew fully supports.
-
Using Athena or other AWS analytics services that have better support for Iceberg tables if your workflow allows.
If you need to specifically use DataBrew with these datasets, you might need to contact AWS Support for further assistance, as this appears to be a service limitation rather than a configuration issue on your end.
Sources
Unable to view Iceberg table data in Athena despite successful Glue crawl | AWS re:Post
Does Kinesis Firehose destination support Glue table format of type Iceberg | AWS re:Post
answered a year ago
Relevant content
asked 3 years ago
asked 4 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 months ago
