By using AWS re:Post, you agree to the Terms of Use

Questions tagged with AWS Database Migration Service

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Trying to copy a postgres instance from a local postgres server to an RDS instance via DMS

`Greetings, thanks for looking at my question.` I've followed [this AWS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html) and set up a VPC, created a replication instance on DMS and set up the database server (it's on an EC2 instance currently, but ultimately we will be using a local server running postgres and the production database from our rack as the source) as the source endpoint and the RDS instance as the target endpoint. I'm trying to set up streaming replication, however, so the information on logical replication and CDC isn't relevant to my setup. ***The goal is to have our production database instance backed up and replicating to an RDS instance, down the road we hope to spin up EC2s and connect them to this RDS instance to scale up as needed. For now though we are just trying to get it set up for proof of concept.*** So I've done everything advised in the docs, but need to copy the database to the RDS instance and start the replication. I created a Database Migration Task in DMS (Full load, ongoing replication), but it fails and doesn't copy over the database or tables. This is the error message on the AWS DMS dashboard for the Migration Task: ``` Last Error Task 'FB24Y2N4*****************************HLB4R3Q' was suspended after 9 successive recovery failures Stop Reason FATAL_ERROR Error Level FATAL ``` Grateful for any advice from anyone who has successfuly set up a similar architecture. Thanks. *** *Edit to clarify: The problem isn't the error per se, it's that I need to find out why the DMS migration task that I created isn't initially copying the instance from my EC2 Postgres source to my AWS RDS instance endpoint and then beginning and maintaining replication. How do I get my database that I'm trying to replicate on RDS copied over?*
1
answers
0
votes
20
views
asked 2 months ago

DMS - Ongoing replication causing duplicate records due to redshift cluster maintenance

Hi, I'm facing an issue with DMS ongoing replication causing a small number of duplicate records due to the target redshift cluster going down for maintenance every week. One of the error messages in the cloudwatch logs is `Failed to start rollback apply transaction`, so it looks like DMS fails to undo a batch, and with `RecoverableErrorCount` set to -1, the process ends up retrying the ongoing replication from the batch it already ran, resulting in the duplicates. Is there a way via the DMS settings to handle this rollback failure issue caused by the cluster maintenance? Or maybe a way to work around the cluster going down for maintenance? Thanks! Below are my settings: Source: Mysql Target: Redshift `{ "ErrorBehavior": { "FailOnNoTablesCaptured": true, "ApplyErrorUpdatePolicy": "SUSPEND_TABLE", "FailOnTransactionConsistencyBreached": true, "RecoverableErrorThrottlingMax": 1800, "DataErrorEscalationPolicy": "SUSPEND_TABLE", "ApplyErrorEscalationCount": 5, "RecoverableErrorStopRetryAfterThrottlingMax": true, "RecoverableErrorThrottling": true, "ApplyErrorFailOnTruncationDdl": false, "DataTruncationErrorPolicy": "SUSPEND_TABLE", "ApplyErrorInsertPolicy": "SUSPEND_TABLE", "EventErrorPolicy": "IGNORE", "ApplyErrorEscalationPolicy": "SUSPEND_TABLE", "RecoverableErrorCount": -1, "DataErrorEscalationCount": 5, "TableErrorEscalationPolicy": "SUSPEND_TABLE", "RecoverableErrorInterval": 5, "ApplyErrorDeletePolicy": "SUSPEND_TABLE", "TableErrorEscalationCount": 5, "FullLoadIgnoreConflicts": true, "DataErrorPolicy": "SUSPEND_TABLE", "TableErrorPolicy": "SUSPEND_TABLE" }, "TTSettings": { "TTS3Settings": null, "TTRecordSettings": null, "EnableTT": false }, "FullLoadSettings": { "CommitRate": 10000, "StopTaskCachedChangesApplied": false, "StopTaskCachedChangesNotApplied": false, "MaxFullLoadSubTasks": 8, "TransactionConsistencyTimeout": 1200, "CreatePkAfterFullLoad": false, "TargetTablePrepMode": "DROP_AND_CREATE" }, "TargetMetadata": { "ParallelApplyBufferSize": 10000, "ParallelApplyQueuesPerThread": 0, "ParallelApplyThreads": 32, "TargetSchema": "", "InlineLobMaxSize": 0, "ParallelLoadQueuesPerThread": 0, "SupportLobs": true, "LobChunkSize": 0, "TaskRecoveryTableEnabled": true, "ParallelLoadThreads": 32, "LobMaxSize": 63, "BatchApplyEnabled": true, "FullLobMode": false, "LimitedSizeLobMode": true, "LoadMaxFileSize": 0, "ParallelLoadBufferSize": 1000 }, "BeforeImageSettings": null, "ControlTablesSettings": { "historyTimeslotInMinutes": 5, "HistoryTimeslotInMinutes": 5, "StatusTableEnabled": true, "SuspendedTablesTableEnabled": true, "HistoryTableEnabled": true, "ControlSchema": "dms_control", "FullLoadExceptionTableEnabled": true }, "LoopbackPreventionSettings": null, "CharacterSetSettings": null, "FailTaskWhenCleanTaskResourceFailed": false, "ChangeProcessingTuning": { "StatementCacheSize": 50, "CommitTimeout": 1, "BatchApplyPreserveTransaction": true, "BatchApplyTimeoutMin": 1, "BatchSplitSize": 0, "BatchApplyTimeoutMax": 30, "MinTransactionSize": 1000, "MemoryKeepTime": 60, "BatchApplyMemoryLimit": 500, "MemoryLimitTotal": 1024 }, "ChangeProcessingDdlHandlingPolicy": { "HandleSourceTableDropped": true, "HandleSourceTableTruncated": true, "HandleSourceTableAltered": true }, "PostProcessingRules": null }`
1
answers
0
votes
74
views
asked 2 months ago