Test Data Management tool for file Anonymization in AWS

Question

Hi All,

I'm looking for a Test Data management (TDM) tool in AWS which can perform below requirements,

1. TDM tool to Connect Production S3 bucket to extract files for anonymization and load it in Test S3 bucket

2. Job scheduled on daily basis to anonymize files in Prod S3 and store files in Test S3 bucket

3. Identify PII columns from S3 file and anonymize it, later these files are loaded in redshift database
4. Data integrity should be maintained between file and database, for example, incremental daily data should able to match the existing mocked PII columns in the redshift database

Kindly let me know how can I achieve above requirements using AWS services

Thanks & Regards,
Aflah

Answer

[AWS Glue Databrew](https://aws.amazon.com/glue/features/databrew/) could help to achieve your objective. It offers different functions to mask data:

1. It can connect to [Amazon S3 and many other data stores ](https://docs.aws.amazon.com/databrew/latest/dg/datasets.html) including Amazon Redshift to read source data and to write the output data.
2. [Databrew Job can be scheduled ](https://docs.aws.amazon.com/databrew/latest/dg/jobs.profile.html) and managed by the service  or integrated with AWS StepFunctions and other workflow services
3. it does include a [PII detection feature](https://aws.amazon.com/about-aws/whats-new/2021/11/aws-glue-databrew-detection-data-masking-transformations/)
4. some of the [data masking techniques ](https://docs.aws.amazon.com/databrew/latest/dg/personal-information-protection.html)that can be used provide repeatable output

you can read more in [this blog post](https://aws.amazon.com/blogs/big-data/introducing-pii-data-identification-and-handling-using-aws-glue-databrew/).

hope this helps.

Answer

Hi, Good question

you could use a service called Comprehend to detect PII and replace it. An example workshop is located at https://github.com/karchit/s3-object-lambda-workshop/tree/Lab2

Test Data Management tool for file Anonymization in AWS

Relevant questions

Load test kinesis data stream consumer

How to know if a CloudTrail event is a Management Event or Data Event?