Issues with ongoing loads/replication/delta loads using DMS

0

Need your help as this falls outside my expertise, here is my customer question:

Currently we are using AWS Database Migration service (DMS) to do one time loads and ongoing loads into red-shift from different Sources.

DMS is great when we doing One time loads. However there are some issues with ongoing loads/ replication/ delta loads.

Sometimes counts are not matching with the error messages log sequence file is missing, even though that particular log file exists on the server.

Solution from support is restart the task. Since we are using this tool in production for daily feeds, restarting the task whenever something issues arises is not a good idea.

Coming to the SQL server, again one time loads are fine with minor problems but live replication needs too many functions on the SQL server like CDC, Replication etc.

Since Our SQL servers are on AlwaysOn tech, adding replication to the server will add load, increases latency and requires more man power to fix the replication issues. Always on is more advanced feature than replication with less maintenance than replication.

Are there any alternative tools to avoid all these problems? something like AWS Glue, AWS data pipeline, any other third party tools.

1개 답변
0
수락된 답변

Almost all the vendors supporting ongoing replication are based on transaction log- a feature that Oracle, SQL Server and MySQL have. This applies to AWS DMS as well as 3rd party tools such as Attunity.

If real time replication is not a requirement then one option is to use a JDBC connector program that can be scheduled to run. The rational of such program is to track changes according to an incremental column in the table- either timestamp or unique sequential value. This is similar to https://redshiftsupport.matillion.com/customer/portal/articles/2955330.

AWS
답변함 5년 전
profile picture
전문가
검토됨 한 달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠