CDC DMS task - removing Op column

0

i have a DMS Replication task that is full+ongoing replication. the source table dont have update or delete operation , only insert for new records. the task is adding Op field with the I (value for insert).

i dont need this field. the initial load csv does not have this field , but the files create after each insert contain it. that cause glue in the next step of the pipeline to create the wront schema and after that some of the values are not in the correct field when quering the data

i have tried using transformation and drop this column , but that didn't work

        {
            "rule-type": "transformation",
            "rule-id": "185351232",
            "rule-name": "185351232",
            "rule-target": "column",
            "object-locator": {
                "schema-name": "%",
                "table-name": "%",
                "column-name": "Op"
            },
            "rule-action": "remove-column",
            "value": null,
            "old-value": null
        },

any idea what i can do to remove it ?

  • using the IncludeOpForFullLoad is not a valid option since the initial load of those table have already done. i am thinking about using lambda to remove this column but i am not sure how good is this idea

avishni
posta un mese fa67 visualizzazioni
3 Risposte
0
  • That rule removes only columns present inherent to source tables.

  • unfortunately, i didnt find any reference in the documentation to removing the first column in the method you mention , only by specifying the column name as i wrote in the question - which does not work

0

Removing opcolumn in cdc is not possible But you can add it in fullload by using eca IncludeOpForFullLoad

AWS
con risposta un mese fa
0
profile picture
ESPERTO
con risposta un mese fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande