How does AWS DMS table selection rules handle overlap or conflict?
Hello! I'm reading through the docs that talk about AWS DMS selection rules and wildcards:
- https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Selections.html
- https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Wildcards.html
I have a use-case where we're currently pulling in a few dozen unnecessary tables in a snapshot load from source (Postgres) to destination (Athena DataLake). I want to exclude MOST tables matching a certain pattern (%_aud
, to be specific), but then there's one table matching that same pattern I want to explicitly keep (let's say its called foo_aud
).
I'd like to not have to write explicit exclusion rules for the ~20 or so other tables, so I'm hoping I can do something like this: (rules are specified in JSON in a cloudformation file)
{
"rules": [
// ...our other selection rules...
{
"rule-type": "selection",
"rule-id": "21",
"rule-name": "excludeaudit",
"object-locator": {
"schema-name": "myschema",
"table-name": "%_aud"
},
"rule-action": "exclude"
},
{
"rule-type": "selection",
"rule-id": "22",
"rule-name": "includefooaudit",
"object-locator": {
"schema-name": "myschema",
"table-name": "foo_aud"
},
"rule-action": "include"
}
]
}
What will happen in the above example? Will DMS correctly exclude all %_aud
tables, but then include foo_aud
? Does the rule ordering matter - e.g. if I swap rules 22 and 21? (How is rule priority managed?) Or is there some other way I can achieve this?
Thank you!
This should work for you. The order of the rule ids in the Mapping Rules of any AWS DMS tasks does not influence the priority unless you use "load-order" in your mapping rules.
Since its only 1 table that you plan to explicitly include, you can try "explicit" as the "rule-action" as well. Something like below:
I would say, go ahead and test both out. Either or both should work.
Relevant questions
DMS service for hive table migration
Accepted Answerasked 2 years agoDMS Sync Issues with nullable column and new columns
asked 5 months agoDoes AWS DMS has any way to touch or modify source tables?
asked 4 days agoAWS DMS performance degradation until stopped and resumed
asked 2 months agoDoes AWS DMS work with Oracle 21c express edition
asked 4 months agoAWS DMS migration task doesn't automatically increment the source table
Accepted Answerasked 2 years agoHow does AWS DMS table selection rules handle overlap or conflict?
asked 4 months agoUsing DMS and SCT for extracting/migrating data from Cassandra to S3
asked 15 days agoMySQL DMS continuous replication with schema changes.
Accepted Answerasked 5 years agoAWS DMS Postgres to OpenSearch LOB handling
asked 4 months ago
Did the shared solution work for you?