DMS Controlling the number of source database connections used
Is this possible?
I have to stagger my ETL processes out because DMS opens up ~34(+7regular) connections on our Postgres source database to complete one transfer task. The table count is 224 for this task/database (one example, there are many databases). If I don't stagger them the total connection count goes way above 100 and then our source server essentially crashes. It simply won't handle that many open connections at once.
Is there any way to control the total connections AWS DMS uses per task job?
Thanks
Edited by: ted1111 on Sep 30, 2021 8:06 AM
Actually by adjusting a couple settings in the task JSON downward I was able to reduce the total resources required on the source database.
See the MaxFullLoadSubTasks and CommitRate settings.
For MaxFullLoadSubTasks it generally corresponds to the number of tables processing at once, although number connections is multiples per table in my case. For CommitRate I lower this to 1000 to conserve all resources on our side, server and network, reduce overall stress.
Edited by: ted1111 on Oct 1, 2021 3:21 AM
Edited by: ted1111 on Oct 1, 2021 9:53 AM
Relevant questions
Do source filters speed up DMS from AWS RDS to S3?
asked 5 months agoDoes AWS DMS has any way to touch or modify source tables?
asked 4 days agoDMS Controlling the number of source database connections used
asked 8 months agoCan DMS be used to extract data from an RDS Oracle Read Replica?
Accepted Answerasked 2 years agoAWS DMS migration task doesn't automatically increment the source table
Accepted Answerasked 2 years agoPostgresRDS 9.4.20 AWS DMS source pglogical shared_preload_libraries param
Accepted Answerasked 3 years agoAWS DMS task is failing on source endpoint timeout
asked a month agoError using Aurora Postgres version 11.8 as DMS source endpoint
asked 2 years agoETL from an Oracle database into RDS Postgres too slow
Accepted Answerasked 2 years agoPhantom Update on Aurora Postgres from AWS DMS Replication from SQL Server Source
asked 3 months ago