Deleting insert and updates from DMS cdc parquet folder

Question

I have a DMS job that writes records from postgres to s3 folder in parquet format. I want, for compliance reasons, to delete records from the parquet file when the records are deleted from source. I know the delete (cdc) row will be null for all columns apart from primary keys but I need the previous insert and updates to be deleted from the parquet files that DMS writes to s3.

Ruchir_A · Answer

Hi,

As I understand, you would like to remove the insert and update records from the parquet files if there is a delete in the source.

Currently, this is not possible with AWS Database Migration Service (DMS). Once a record (insert operation or update operation) is written to the parquet file in S3 bucket, DMS cannot go back and update the parquet file and remove the records.

Deleting insert and updates from DMS cdc parquet folder

Add your answer

Relevant content