Questions on Data lake using DMS
A customer is implementing a data lake and i am assessing using DMS for pulling data into s3 from on an premise MS SQL DB. As part of this i would like to understand the following points on DMS:
- Other users of DMS - widely used?
- Examples of using DMS with SQL Server Change Tracking
- Volumetrics - how much can it handle
- Running over VPN - any issues that you're aware of?
- Known issues/limitations of DMS (big binary objects etc)
- What ports/access (accounts) does it require for the on prem SQL server
- What is the overhead on the on prem SQL server?
Other users of DMS - widely used? yes, > 100,000 moved to aws using DMS so far
Volumetrics - how much can it handle? Never hit a limit of not keeping up yet.
Running over VPN - any issues that you're aware of? No, other than if using CDC then the CDC change tables could get larger if there’s not enough bandwidth to send the data to DMS instance in the cloud.
Known issues/limitations of DMS (big binary objects etc)? All documented: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html
What ports/access (accounts) does it require for the on prem SQL server? 1433, sysadmin access to read the logs and CDC tables
What is the overhead on the on prem SQL server? Depends. Transactional repl will add more info to the transaction log, will produce bigger logs. CDC tables will take more space in the customer database, and add some sqlagent jobs to do cleanup.
Data ingestion using DMS or Glue?Accepted Answerasked 2 years ago
About DMS migration from ms-sql to dynamodbasked 24 days ago
DMS CDC for SQL Server - ReplicationAccepted AnswerEXPERTasked 2 years ago
Questions on Data lake using DMSAccepted Answerasked 4 years ago
DMS charges for target S3 Bucket PUTsAccepted Answerasked 2 years ago
Data ingestion from Oracle Cloud into S3Accepted Answerasked 4 years ago
Copying data from sql server to snowflake with AWS GLUEasked a month ago
Is there a service to move a database from MSSQL to MySQL (both hosted on AWS)asked 5 months ago
data lake - data ingestion methodsAccepted Answerasked 4 years ago
Sagemaker and Data on DatabasesAccepted Answerasked 4 years ago