Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Does Data quality work with Apache Iceberg tables

1

I get an error when trying the recommended rules for an iceberg table: InvalidInputException (status: 400): Invalid table definition: Parameter "table_type" in table stg_l1_stf_lineitemmap is set to "iceberg" but storage location doesn't conform to the format "s3://<warehouse-dir>/<glue database>/<glue table id>". To specify your warehouse directory, pass it in the request through AdditionalOptions={WarehouseDirectory=<your-warehouse-dir>}

Also get this error when running DQ rule through a glue job python script that works for other tables: An error occurred while calling o81.getDynamicFrame. Reads and writes using Lake Formation permissions are not supported for iceberg tables.

Topics
Analytics
Tags
AWS Glue
Language
English
asked 2 years ago1.1K views
2 Answers
0

Please refer the below documentation, data quality is supported:
[1] AWS Glue Data Quality - General availability: new features - https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html#data-quality-release-notes-ga

The following new features are available with the general availability of AWS Glue Data Quality:

  • Support for Apache Iceberg, Apache Hudi and Delta Lake
AWS
answered 2 years ago
  • neal_li
    2 years ago

    Then why am i getting an error "An error occurred while calling o81.getDynamicFrame. Reads and writes using Lake Formation permissions are not supported for iceberg tables."

  • CF Plus - De Bortoli
    9 months ago

    Update on Glue Data Quality and Apache Iceberg Tables We are aware that some users are experiencing issues with Glue Data Quality when creating rules on Iceberg tables managed through Lake Formation. While the recent release mentioned here (https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html#data-quality-release-notes-ga) states that Data Quality works with Apache Iceberg tables, this does not yet include support for Iceberg tables managed through Lake Formation. We are working to address this issue and will provide an update as soon as possible. In the meantime, if you are using Lake Formation to manage your Iceberg tables, you may need to use alternative methods to ensure data quality.

    My log: java.lang.UnsupportedOperationException: Reads and writes using Lake Formation permissions are not supported for iceberg tables.

0

+1 also me receiving "Reads and writes using Lake Formation permissions are not supported for Hudi tables." please cut a ticket to investigate Data Catalog DQ Jobs and their compatibility on Tables with Open-source data lake frameworks

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content