How to apply Data Quality rules on Hudi tables and show results in Amazon DataZone?

Since AWS Glue Data Quality doesn't directly support Hudi tables managed by Lake Formation, you'll need to implement an alternative approach. Here's how you can apply data quality rules to your Hudi tables and display the results in Amazon DataZone:

1. Implement an external data quality solution: You can use Amazon EMR Serverless with a framework like PyDeequ to create custom data quality rules for your Hudi tables. This approach allows you to define tailored data quality checks such as schema validation, null checks, and data range checks.

2. Create and execute custom data quality rules:

• Develop your data quality rules using PyDeequ or a similar framework
• Run these rules using EMR jobs that can access your Hudi tables through Athena
• Generate data quality scores based on the results of your rules

3. Integrate with Amazon DataZone:

• Amazon DataZone provides APIs for importing data quality scores from external systems
• Create a script (e.g., in Python) that takes the results from your EMR job and formats them appropriately
• Use the Amazon DataZone APIs to post the data quality scores to DataZone
• This script can be executed as part of your EMR workflow or as a separate step after the data quality checks

4. Configure permissions:

• Ensure Amazon DataZone has the necessary permissions to access the AWS Glue database managed through Lake Formation
• This may involve configuring Lake Formation permissions specifically for Amazon DataZone

5. Visualize in DataZone:

• Once the integration is set up, you'll be able to view the data quality scores for your Hudi tables directly in the Amazon DataZone console
• This provides visibility to your stakeholders without requiring them to access the underlying data quality tools

This approach allows you to overcome the limitations of AWS Glue Data Quality for Hudi tables while still providing integrated data quality information in Amazon DataZone. Remember to regularly review and update your data quality rules as your data and requirements evolve.
Sources
Data Quality Rules for Hudi Tables Managed by Lake Formation in AWS DataZone | AWS re:Post
Implement data quality checks on Amazon Redshift data assets and integrate with Amazon DataZone | AWS Big Data Blog