By using AWS re:Post, you agree to the Terms of Use
/How does AWS DMS table selection rules handle overlap or conflict?/

How does AWS DMS table selection rules handle overlap or conflict?

0

Hello! I'm reading through the docs that talk about AWS DMS selection rules and wildcards:

  1. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Selections.html
  2. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Wildcards.html

I have a use-case where we're currently pulling in a few dozen unnecessary tables in a snapshot load from source (Postgres) to destination (Athena DataLake). I want to exclude MOST tables matching a certain pattern (%_aud, to be specific), but then there's one table matching that same pattern I want to explicitly keep (let's say its called foo_aud).

I'd like to not have to write explicit exclusion rules for the ~20 or so other tables, so I'm hoping I can do something like this: (rules are specified in JSON in a cloudformation file)

{
    "rules": [
// ...our other selection rules...
        {
            "rule-type": "selection",
            "rule-id": "21",
            "rule-name": "excludeaudit",
            "object-locator": {
                "schema-name": "myschema",
                "table-name": "%_aud"
            },
            "rule-action": "exclude"
        },
        {
            "rule-type": "selection",
            "rule-id": "22",
            "rule-name": "includefooaudit",
            "object-locator": {
                "schema-name": "myschema",
                "table-name": "foo_aud"
            },
            "rule-action": "include"
        }
    ]
}

What will happen in the above example? Will DMS correctly exclude all %_aud tables, but then include foo_aud? Does the rule ordering matter - e.g. if I swap rules 22 and 21? (How is rule priority managed?) Or is there some other way I can achieve this?

Thank you!

  • Did the shared solution work for you?

1 Answers
0

This should work for you. The order of the rule ids in the Mapping Rules of any AWS DMS tasks does not influence the priority unless you use "load-order" in your mapping rules.

Since its only 1 table that you plan to explicitly include, you can try "explicit" as the "rule-action" as well. Something like below:

{ "rules": { "rule-type": "selection", "rule-id": "1", "rule-name": "1", "object-locator": { "schema-name": "NewCust", "table-name": "Customer" }, "rule-action": "explicit" } }

I would say, go ahead and test both out. Either or both should work.

answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions