AWS Glue Data Quality with Iceberg

Question

Hallo, 
I would like to add some data quality rules to my existing tables which are in Iceberg format. (Created via Glue).
After setting up the DQ rule got the following error message:
```

Error encountered
InvalidInputException (status: 400): Invalid table definition: Parameter "table_type" in table [REDACTED] is set to "iceberg" but storage location doesn't conform to the format "s3:////". To specify your warehouse directory, pass it in the request through AdditionalOptions={WarehouseDirectory=}
```
We had to use a custom location of the warehouse directory.
But how can I specify the warehouse directory? I cannot see any place where can I paste that info in the Data Quality module.
Thanks in advance!

Gonzalo Herreros · Answer

I don't think you can specify that property but you shouldn't need either. 
Check the table location property, as long as it points to a directory (but without a trailing /). 
It expects something like this:  
s3://mybucket/myWH/mydb.db/table1   
s3://mybucket/somePath/mydb.db/table1  
(In practice it will consider everything up to the db folder as the Warehouse dir)

AWS Glue Data Quality with Iceberg

Ajoutez votre réponse

Contenus pertinents