AWD DMS setting CDC file size

Question

Hi Everyone,

I configured an AWS DMS to migrate data from an AWS Managed Oracle database to a S3 Bucket.

The CDC has been creating file greater than 32MB.  I was expecting to have file with at most 32MB due to the `cdcMinFileSize`. In my bucket there are files greater than 200MB.

**cdcMaxBatchInterval** - The maximum interval length condition, defined in seconds, to output a file to Amazon S3. The default value is 60 seconds.

**cdcMinFileSize** - The minimum file size condition, defined in KB, to output a file to Amazon S3. The default value is 32000 KB.

**WriteBufferSize** - The size, in KB, of the in-memory file write buffer used when generating .csv files on the local disk at the AWS DMS replication instance. The default value is 1000 KB.

I was wondering if my oracle database committed a large transaction that lead to this scenario. But during the processing on the Replication Instance would break the file into chunks to fit the 32MB?

Thanks in advance.

Answer

cdcMinFileSize defines the least or smallest file that AWS DMS will create on S3. If you are seeing a 200 MB file, its expected. Bottom line, with the default setting of 32000 (KB) for cdcMinFileSize, you will not see any files on S3 smaller than 32000 KB.

AWD DMS setting CDC file size

Relevant content