Record Counts in Parquet and SQL

Question

Should the parquet files created by DMS contain the same number of records as the SQL change table? Very helpful to know in order to compare records in SQL log to that in the S3 parquet. .

Answer

Hi,
If you look at section CdcInsertsOnly of https://docs.aws.amazon.com/dms/latest/APIReference/API_S3Settings.html, it clearly says that UPDATEs, INSERTs and DELETEs are reported in parquet files with one row per change.

```
By default (the false setting), the first field in a .csv or .parquet record contains
 the letter I (INSERT), U (UPDATE), or D (DELETE). These values indicate whether 
the row was inserted, updated, or deleted at the source database for a CDC load 
to the target.
```

So, all changes are clearly reported.

I personally use not with Parquet but CSV files and it works fine.

Best,
Didier

Record Counts in Parquet and SQL

Contenuto pertinente