Controlling Frequency of Writes to S3 Target With DMS

0

Hey All,

I am trying to use DMS for a CDC task (PostgreSQL RDS -> S3). I want to control how often updates are written to the S3 target. I tested a number of settings by changing the “Task Settings” param.

BatchApplyEnabled: true, BatchApplyTimeoutMin/BatchApplyTimeoutMax: 3
BatchApplyEnabled: true, BatchApplyTimeoutMin/BatchApplyTimeoutMax: 3, MinTransactionSize: 0
BatchApplyEnabled: false, MinTransactionSize: 0

None of these seemed to have any impact on the time for a change record to appear in S3. All were tested via a single insert. Regardless of setting, it took ~1 minute for the new parquet file to appear in S3.

Any ideas on how to control this kind of thing with an S3 target? Thanks.

Edited by: Weves on May 4, 2020 3:37 PM

Weves
asked 4 years ago870 views
4 Answers
0

Hello Weves,

Thanks for reaching out and it seems you are trying to control CDC with S3 target. Please note that batch apply mode is not applicable or supported with S3 target.

You can use below internal parameters ( as target endpoint extra connection attributes) to control the way changes are applied on target during replication.

cdcMaxBatchInterval default value 60 seconds
cdcMinFileSize default value 32 MB

Please change values for above parameters and it will help you change how csv file is generated during CDC.

I hope this information helps... I will try to get this parameter documented on our public docs..

Thanks,
Mahesh Kansara

Edited by: MaheshKansara on May 4, 2020 4:58 PM

answered 4 years ago
0

Thanks. The cdcMaxBatchInterval param worked pretty well for us.

Weves
answered 4 years ago
0

What are the measurement units for these params? And are they documented anywhere in AWS docs yet?? Can't find them.
cdcMaxBatchInterval - Is this in seconds?
cdcMinFileSize - Is this in MB? KB?

answered 3 years ago
0

Hi pcguy166,

You're correct, cdcMaxBatchInterval is in seconds, and cdcMinFileSize is in MB. This is documented in the "Extra connection attributes when using Amazon S3 as a target for AWS DMS" section of the User Guide, which you can access via the link below.

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html#CHAP_Target.S3.Configuring

Best,
Sean

AWS
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions