DMS not replicating addition of new columns to source

Question

Have created a DMS task to replicate data from MongoDB to S3 as parquet files. The updates, inserts, deletes are working fine and replicating in target, but the new columns added to source are not reflecting in the S3 file generated from DMS. The file contains the same schema as source when DMS ran a full load

What all settings should I do in the task to reflect these alter statements of adding column to target?

Answer

this is expected behaviour

mongodb as source for dms can be configured two ways

document mode and table mode

in document mode whole row irrespective is migrated as single column along with _id

```
> db.myCollection.find()
{ "_id" : ObjectId("5a94815f40bd44d1b02bdfe0"), "a" : 1, "b" : 2, "c" : 3 }
{ "_id" : ObjectId("5a94815f40bd44d1b02bdfe1"), "a" : 4, "b" : 5, "c" : 6 }
```

to

```
oid_id	                        _doc
5a94815f40bd44d1b02bdfe0	{ "a" : 1, "b" : 2, "c" : 3 }
5a94815f40bd44d1b02bdfe1	{ "a" : 4, "b" : 5, "c" : 6 }
```
thus during replication it will bring forward all columns as one doc

while in table mode (seems to be your case)

target columns are mapped as

```
oid_id	                        a	b	c
5a94815f40bd44d1b02bdfe0	1	2	3
5a94815f40bd44d1b02bdfe1	4	5	6
```
as mentioned at https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MongoDB.html#:~:text=To%20create%20the%20target%20columns
the target columns are limited during full load and are as per the docs being investigated (default 1000) for setting target columns 
thus any further additions of columns during replication wont be replicated

Say the source following record is added in replication

`{ "_id" : ObjectId("4a94815f405d45d1b02bdf51"), "a" : 3, "d" : 5, "c" : 7 }`

target would have

```
oid_id	                        a	b	c
4a94815f405d45d1b02bdf51	3		7
```

i hope the explanation is clear

DMS not replicating addition of new columns to source

相关内容