By using AWS re:Post, you agree to the Terms of Use
/DMS not replicating addition of new columns to source/

DMS not replicating addition of new columns to source

0

Have created a DMS task to replicate data from MongoDB to S3 as parquet files. The updates, inserts, deletes are working fine and replicating in target, but the new columns added to source are not reflecting in the S3 file generated from DMS. The file contains the same schema as source when DMS ran a full load

What all settings should I do in the task to reflect these alter statements of adding column to target?

1 Answers
0

this is expected behaviour

mongodb as source for dms can be configured two ways

document mode and table mode

in document mode whole row irrespective is migrated as single column along with _id

> db.myCollection.find()
{ "_id" : ObjectId("5a94815f40bd44d1b02bdfe0"), "a" : 1, "b" : 2, "c" : 3 }
{ "_id" : ObjectId("5a94815f40bd44d1b02bdfe1"), "a" : 4, "b" : 5, "c" : 6 }

to

oid_id	                        _doc
5a94815f40bd44d1b02bdfe0	{ "a" : 1, "b" : 2, "c" : 3 }
5a94815f40bd44d1b02bdfe1	{ "a" : 4, "b" : 5, "c" : 6 }

thus during replication it will bring forward all columns as one doc

while in table mode (seems to be your case)

target columns are mapped as

oid_id	                        a	b	c
5a94815f40bd44d1b02bdfe0	1	2	3
5a94815f40bd44d1b02bdfe1	4	5	6

as mentioned at https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MongoDB.html#:~:text=To%20create%20the%20target%20columns the target columns are limited during full load and are as per the docs being investigated (default 1000) for setting target columns thus any further additions of columns during replication wont be replicated

Say the source following record is added in replication

{ "_id" : ObjectId("4a94815f405d45d1b02bdf51"), "a" : 3, "d" : 5, "c" : 7 }

target would have

oid_id	                        a	b	c
4a94815f405d45d1b02bdf51	3		7

i hope the explanation is clear

answered a month ago
  • So there is no way to add new columns in target when the source table is altered-new column added using the table-mode? Will this(new column additions) reflect in document-mode then?

  • thats correct being dynamic nature of mongodb its not possible in table mode. in document mode it will bring whole row or rather document as one column irrespective of no of columns involved in it.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions