DMS Migration Task(only CDC) for Ongoing Replication Type, Migration progress struck at 0%

0

I have created CDC DMS Task to Replicate oracle RDS Data between two environments. I observed when we are having multiple tables in Mapping rules, When I create task,Migration progress is struck at 0% .When I stop and resume the task.I am ablo to see migration progress as 100%. Usually in what scenarios, do we face it? I dont see any errors as well. We observed that, When Migration Progress is 0% , it will not actually replicate the data..only after progress shows 100% it is replicating data.

Would having triggers in Target table cause this? Usually sometimes when we create the task itself we are able to see e migration progress as 100% but sometimes it is struck at 0% and wont move to 100% until we stop and resume the task.

Source : Oracle RDS Target : Oracle RDS in different env Replication Type: Replication Ongoing only( No Full load) MigrationProgress

Below is how our metrics looks like : Enter image description here Enter image description here

Enter image description here

Enter image description here

asked 3 months ago178 views
1 Answer
0

That issue can occur due to various reasons.

  1. CDC Load Balancing: AWS DMS uses a load balancing mechanism for Change Data Capture (CDC) tasks to distribute the load across multiple threads. If the source database doesn't generate enough change events initially, the progress might show as 0% until enough changes accumulate. Stopping and resuming the task can trigger the load balancing mechanism and start the replication process.

  2. Table Metadata Retrieval: Before starting the CDC replication, DMS retrieves the table metadata from the source and target databases. If there are a large number of tables or complex table structures, this process can take some time, during which the migration progress might show as 0%.

  3. Triggers on Target Tables: As you mentioned, triggers on the target tables can potentially cause issues with the CDC replication process. DMS might need to perform additional checks or handling when dealing with triggers, which could delay the migration progress from updating initially.

  4. Large Transaction Logs: If the source database has a significant amount of transaction logs or change events to process initially, DMS might take some time to catch up and start showing progress. This delay could be more pronounced if there are also other factors like complex table structures or a large number of tables.

  5. Network Latency: High network latency between the DMS replication instance and the source or target databases can sometimes cause delays in the progress updating. This is more likely to occur in cross-region or cross-account setups.

  6. Database Load: If the source or target databases are under heavy load, it might impact the CDC replication process and cause delays in the progress updating initially.

To mitigate these issues, you can try the following:

  1. Increase CDC Max Batch Size: You can increase the maxBatchSize parameter in the DMS task settings to allow DMS to process larger batches of changes, which might help in scenarios with high change volume or large transaction logs.

  2. Use Parallel Load Threads: If you have a large number of tables, consider increasing the number of parallel load threads in the DMS task settings to speed up the initial metadata retrieval process.

  3. Optimize Triggers: Review and optimize any triggers on the target tables to minimize their impact on the CDC replication process.

  4. Monitor Database Load: Monitor the source and target database loads during the CDC replication process and consider adjusting resources or scheduling the migration during off-peak hours if necessary.

  5. Check Network Latency: If you're experiencing high network latency, consider moving the DMS replication instance closer to the source and target databases or optimizing the network connectivity.

profile picture
EXPERT
answered 3 months ago
  • Thank you for the response, We have constraints in DB Tables and also we need to follow order of insertions/updations. As per this we cant use Parallel Load Threads and Batch Processing right? Regarding CDC Load Balancing you have mentioned in 1st point, we tried inserting changes in source when Migration progress is 0% but still it is struck there until we stop and resume the task. Do we have any option to validate if this behavior is due to CDC Load Balancing. We have 6 Migration tasks out of which 5 are struck at 0%, Updated Query with attaching how our metrics looks when they are struck at 0%.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions