DMS not replicating addition of new columns to source
Have created a DMS task to replicate data from MongoDB to S3 as parquet files. The updates, inserts, deletes are working fine and replicating in target, but the new columns added to source are not reflecting in the S3 file generated from DMS. The file contains the same schema as source when DMS ran a full load
What all settings should I do in the task to reflect these alter statements of adding column to target?
this is expected behaviour
mongodb as source for dms can be configured two ways
document mode and table mode
in document mode whole row irrespective is migrated as single column along with _id
> db.myCollection.find()
{ "_id" : ObjectId("5a94815f40bd44d1b02bdfe0"), "a" : 1, "b" : 2, "c" : 3 }
{ "_id" : ObjectId("5a94815f40bd44d1b02bdfe1"), "a" : 4, "b" : 5, "c" : 6 }
to
oid_id _doc
5a94815f40bd44d1b02bdfe0 { "a" : 1, "b" : 2, "c" : 3 }
5a94815f40bd44d1b02bdfe1 { "a" : 4, "b" : 5, "c" : 6 }
thus during replication it will bring forward all columns as one doc
while in table mode (seems to be your case)
target columns are mapped as
oid_id a b c
5a94815f40bd44d1b02bdfe0 1 2 3
5a94815f40bd44d1b02bdfe1 4 5 6
as mentioned at https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MongoDB.html#:~:text=To%20create%20the%20target%20columns the target columns are limited during full load and are as per the docs being investigated (default 1000) for setting target columns thus any further additions of columns during replication wont be replicated
Say the source following record is added in replication
{ "_id" : ObjectId("4a94815f405d45d1b02bdf51"), "a" : 3, "d" : 5, "c" : 7 }
target would have
oid_id a b c
4a94815f405d45d1b02bdf51 3 7
i hope the explanation is clear
thats correct being dynamic nature of mongodb its not possible in table mode. in document mode it will bring whole row or rather document as one column irrespective of no of columns involved in it.
Relevant questions
How to use transformation rule in DMS task to perform replace operation(remove space) for all columns
asked a month agoRDS/DynamoDB/DMS questions
Accepted Answerasked 3 years agoDMS for Replicating Data in Oracle Application Container
asked 4 months agoDMS not replicating addition of new columns to source
asked a month agoDo source filters speed up DMS from AWS RDS to S3?
asked 6 months agoDMS - How to retain non-ASCII values when running DMS task from MySQL source to Redshift target
asked 3 months agoDMS Migration MongoDB source and RDS Postgres as destination
Accepted Answerasked 21 days agoAWS DMS CDC task with Mongodb as source fails
asked a month agotable not replicating in DMS from MySQL RDS to Redshift RDS
asked 3 years agoApplying column filters in DMS Task for source MongoDB and target Postgres
asked 20 days ago
So there is no way to add new columns in target when the source table is altered-new column added using the table-mode? Will this(new column additions) reflect in document-mode then?