DMS documentdb cdc loads full, but it doesn't replicate/load deltas

Question

Hi All,

I have a full-load-and-cdc task from documentdb to opensearch cluster. It loads full sync ok, but it doesn't sync deltas/new records. Here's what I see in logs:

```
2024-12-17T15:24:39.000Z
2024-12-17T15:24:39 [TASK_MANAGER ]I: Loading finished for table 'carme'.'widgets' (Id = 1) by subtask 1. 115524 rows affected. Time taken: 2397 seconds (replicationtask.c:2971)
2024-12-17T15:24:39.000Z
2024-12-17T15:24:39 [SOURCE_UNLOAD ]I: Destroying mongoc client: '23050620334048' (mongodb_imp.c:1470)
2024-12-17T15:24:39.000Z
2024-12-17T15:24:39 [SOURCE_CAPTURE ]I: Destroying mongoc client: '23050420600272' (mongodb_imp.c:1470)
2024-12-17T15:24:39.000Z
2024-12-17T15:24:39 [TASK_MANAGER ]I: Subtask #1 ended (replicationtask_util.c:595)
2024-12-17T15:25:41.000Z
2024-12-17T15:25:41 [TARGET_APPLY ]I: No records received to load or apply on target , waiting for data from upstream (streamcomponent.c:2014)
2024-12-17T15:25:47.000Z
2024-12-17T15:25:47 [SORTER ]I: No records received to load or apply on target , waiting for data from upstream (streamcomponent.c:2014)
2024-12-17T15:25:47.000Z
2024-12-17T15:25:47 [SOURCE_CAPTURE ]I: Sent captured record 0 to internal queue from Source (streamcomponent.c:3034)
2024-12-17T15:27:45.000Z
2024-12-17T15:27:45 [SORTER ]I: Task is running (sorter.c:761)
2024-12-17T15:28:41.000Z
2024-12-17T15:28:41 [TARGET_APPLY ]I: No records received to load or apply on target , waiting for data from upstream (streamcomponent.c:2014)
```
What might be wrong ?

re:Post Agent · Accepted Answer

Based on the logs you've provided and the issue you're describing, there are a few potential reasons why your AWS Database Migration Service (DMS) task is not replicating changes (deltas) after the initial full load from DocumentDB to OpenSearch:

1. Change Streams: Ensure that change streams are enabled on your DocumentDB cluster. DMS uses the operations log (oplog) to capture changes during ongoing replication. If change streams are not enabled, DMS won't be able to capture and replicate changes.

2. Oplog Retention: Check if the oplog in DocumentDB is large enough to retain changes for the duration between DMS reads. If DocumentDB flushes out the records from the oplog before DMS reads them, your tasks will fail to replicate changes. It's recommended to size the oplog to retain changes for at least 24 hours.

3. Task Configuration: Verify that your DMS task is configured for both full load and CDC (Change Data Capture). Make sure the "Migration type" is set to "Migrate existing data and replicate ongoing changes" in your task settings.

4. Replication Instance Storage: Check if your replication instance has sufficient storage. If the replication instance storage reaches 90% of capacity, CDC ongoing changes can be paused. This could explain why you're seeing "No records received to load or apply on target" in your logs.

5. Network Connectivity: Ensure there are no network issues between your DMS replication instance and the DocumentDB source.

6. Permissions: Verify that the user account used for the DMS source endpoint has the necessary permissions to read the oplog and capture changes.

To troubleshoot:
1. Check the DMS task status and metrics in the AWS console for any errors or warnings.
2. Verify that changes are actually occurring in your source DocumentDB after the full load.
3. Consider increasing the log verbosity of your DMS task to get more detailed information.
4. If the issue persists, you may need to recreate the task or consider scaling up your replication instance if resource constraints are the problem.

If none of these solve the issue, you may need to contact AWS support for further assistance.  
**Sources**  
[Using Amazon DocumentDB (with MongoDB compatibility) as a source for AWS DMS - AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.DocumentDB.html)  
[Resolve CDC task paused issue in AWS DMS | AWS re:Post](https://repost.aws/knowledge-center/dms-replication-task-paused)  
[Upgrading your Amazon DocumentDB cluster using AWS Database Migration Service - Amazon DocumentDB](https://docs.aws.amazon.com/documentdb/latest/developerguide/docdb-migration.versions.html)

sokoow · Answer

it was indeed change streams missing: https://docs.aws.amazon.com/documentdb/latest/developerguide/change_streams.html

DMS documentdb cdc loads full, but it doesn't replicate/load deltas

Add your answer

Relevant content