RDS/DynamoDB/DMS questions

Question

Is it possible to replicate changes only from RDS to an already created DynamoDB Table?   And have one of the the DynamoDB attributes not be mapped from the RDS columns?  
---

I am trying to migrate one of the tables in RDS to dynamoDB table using DMS.  
Things to note:

1. I want to replicate only the ongoing changes.
2. The DynamoDB table is already created.
3. One of the columns in the new DynamoDB will be completely new, i.e, it won't be derived from the RDS table.

I want to understand:

1. Whether the migration is possible in the above scenario.
2. The risks involved.
3. The latency we might see.
4. Is it guaranteed that all of the data will be migrated?

Accepted Answer

> 1) I want to replicate only the ongoing changes.

this is possible while creating the task make sure you select task type as " replicate ongoing changes"

> 2) The DynamoDB table is already created.

This is also possible. 
DMS supports ddb as target with two types for target table preparation mode
1. drop and recreate
2. do nothing
Make sure customer selects do_nothing 
this setting will search for required table on target if not found it will create one but if it exists it will use the same.

> 3) One of the columns in the new DynamoDB will be completely new, i.e, it won't be derived from the RDS table.

dynamodb table could have more of columns and dms will update data only for the columns in source.

I want to understand: 
1) Whether the migration is possible in the above scenario.

it is possible

2) The risks involved.

any source 
and target DDB related limitations needs to checked.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.DynamoDB.html#CHAP_Target.DynamoDB.Limitations

3) The latency we might see.

depends on the changes happening on source and whether the type of instance class used for the replication instance is able to handle it.

4) Is it guaranteed that all of the data will be migrated?

as long as dms doesnt hit any limitation it should work fine.

Answer

Some more comments to add to above Answer

**2.  The risks involved.  **
LOE risk, if you have never done this before.  It may not be as simple as it may seem depending on your use case.  Ongoing changes (CDC) may require changes to your source DB (e.g. enabling logical replication on Postgres) which requires a restart, and may cause issues on source DB (replication slot eating up more and more storage on source, this can be resolved by deleting the replication slot).  Highly recommend doing a perf test in a non-production environment.  You can also do a perf test based on production data by restoring a snapshot to a new RDS and using that as the source.

**4.  Is it guaranteed that all of the data will be migrated? **
 Not necessarily, if you have duplicates, or if there are performance issues.  If you have more than one record in your source which map to the same sort key in your target, duplicates will be dropped non-deterministically.  Therefore you must remove duplicates prior to migration.  Also, DMS does not support joins so any joins must be done beforehand, by creating a premerged table (or possibly a materialized view may work, not sure on that one).  I cover this limitation and others in this video https://www.youtube.com/watch?v=0kzUIJjo9PY

RDS/DynamoDB/DMS questions

Is it possible to replicate changes only from RDS to an already created DynamoDB Table? And have one of the the DynamoDB attributes not be mapped from the RDS columns?

Conteúdo relevante