Data Quality using PyDeequ

Question

Hi, Does anyone use PyDeequ for large enterprises. I am exploring this library and have the below questions:
1) Looking at the github repo it doesnt seem like it is actively udated. ALso, it supoorts Spark 3.0.0 but not later versions.
2) Some of the apis didnt work(for complex examples). I dont know if there is any Amazon support.
3) Also the scala version(deequ) is more up to date than the python version(PuDeequ). s is there a plan to sunset the PyDeequ version
4) Should I use this for large enterprise data validation framework or there are any other alternate tools. Kindly advise.

Thank you!

Answer

Hi

To answer question '4' - I would recommend you take a look at AWS Glue DataBrew.  Not only is it a fully managed service, but you'll also find that it has a better velocity of new features & updates as its supported by the AWS Glue team.

Thanks

Nick

Data Quality using PyDeequ

Relevant questions

Is it possible to use CloudFront Functions with CloudFormation, and have the source code in an external file?

How do I upload a checkpoint file to my Github repo

Data Quality Framework in AWS

How To Get Bad Records Using AWS Pydeequ - Data Quality Checks

Does the 1.7 Setup Assistant doesn't seem to be working properly... Is it working for anyone else?

GroundTruth text labelling - hide data columns, and methods of quality control

Data Quality using PyDeequ

I am trying to write an ETL job to the Data Catalog but its writing the Headers as Data

Small Scale VOD Streaming - Am I looking in the right place?

I need to read S3 data, transform and put into Data Catalog. Should I be using a Crawler?