Deequ - how to compute file freshness

Hi,

Did you envision to use AWS Glue Data Quality from the AWS Glue Data Catalog to obtain the metrics that you need ?

See https://aws.amazon.com/blogs/big-data/getting-started-with-aws-glue-data-quality-from-the-aws-glue-data-catalog/ for all details.

Best,

Didier

To compute file freshness for your parquet files stored in S3 using pyDeequ, you can leverage the S3 file metadata, specifically the last modified time of the files or folders. Since your data doesn't have a created timestamp, using the folder timestamp is a good alternative approach.

Here's how you can achieve this:

1. First, you'll need to read the S3 file metadata to get the last modified time. You can do this using the AWS SDK for Python (boto3).

2. Once you have the timestamp, you can add it as a new column to your DataFrame.

3. Then, you can use pyDeequ to define and compute a DataFreshness metric based on this new column.

Here's a general outline of the process:

1. Get the S3 file metadata:

   import boto3
   
   s3 = boto3.client('s3')
   response = s3.head_object(Bucket='your-bucket-name', Key='path/to/your/folder/')
   last_modified = response['LastModified']

2. Read your parquet files and add the timestamp:

   from pyspark.sql.functions import lit
   from awsglue.context import GlueContext
   from pyspark.context import SparkContext
   
   sc = SparkContext.getOrCreate()
   glueContext = GlueContext(sc)
   spark = glueContext.spark_session
   
   df = spark.read.parquet("s3://your-bucket-name/path/to/your/folder/")
   df = df.withColumn("file_ts", lit(last_modified))

3. Use pyDeequ to compute the DataFreshness metric:

   from pydeequ.checks import *
   from pydeequ.verification import *
   
   check = Check(spark, CheckLevel.Warning, "Data Freshness Check")
   check.hasDataFreshness("file_ts", lambda x: x < timedelta(hours=24), "Data should be less than 24 hours old")
   
   verificationResult = VerificationSuite(spark) \
       .onData(df) \
       .addCheck(check) \
       .run()

This approach allows you to measure data freshness based on the folder timestamp, even when your data doesn't contain a created timestamp. The DataFreshness check in pyDeequ will evaluate whether the files in your S3 folder have been modified within the specified time frame (in this example, within the last 24 hours).

Remember to adjust the time threshold in the DataFreshness check according to your specific requirements. Also, ensure you have the necessary permissions to access the S3 bucket and read the file metadata.

Sources
DataFreshness - AWS Glue
Using the Parquet format in AWS Glue - AWS Glue