By using AWS re:Post, you agree to the Terms of Use
/DMS Controlling the number of source database connections used/

DMS Controlling the number of source database connections used

0

Is this possible?

I have to stagger my ETL processes out because DMS opens up ~34(+7regular) connections on our Postgres source database to complete one transfer task. The table count is 224 for this task/database (one example, there are many databases). If I don't stagger them the total connection count goes way above 100 and then our source server essentially crashes. It simply won't handle that many open connections at once.

Is there any way to control the total connections AWS DMS uses per task job?

Thanks

Edited by: ted1111 on Sep 30, 2021 8:06 AM

1 Answers
0

Actually by adjusting a couple settings in the task JSON downward I was able to reduce the total resources required on the source database.

See the MaxFullLoadSubTasks and CommitRate settings.

For MaxFullLoadSubTasks it generally corresponds to the number of tables processing at once, although number connections is multiples per table in my case. For CommitRate I lower this to 1000 to conserve all resources on our side, server and network, reduce overall stress.

Edited by: ted1111 on Oct 1, 2021 3:21 AM

Edited by: ted1111 on Oct 1, 2021 9:53 AM

answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions