DMS Validation not running under some circumstances

1

I'm currently facing an issue with an AWS DMS task that involves validating data. My source database is Postgres, and the target for the data is an S3 bucket. Although I have enabled validation for the task, it's failing to run under certain conditions.

When my DMS task includes a large number of tables (around a dozen or more), the validation status remains stuck at "Pending validation". The logs provide minimal information, indicating an issue with the Athena manager not being able to locate the tables, stating they don't exist.

For example, the logs show messages like "S3 Target Validation Reached. Implementation In Progress," followed by "Athena Manager has no stored table name for table with iId 7," which suggests that it's unable to fetch the table name for the validation query.

Interestingly, when the task includes smaller tables, bellow 100k rows, the validation completes successfully. Issues arise when the task includes a large table, over 1-2 million rows. This problem occurs even with two tables, one small with 8k rows, and one big, with 5 million rows. Validation completes successfully for the small table, but won’t even start for the bigger table. When that table is on it’s own in a dms task, the validation completes .

Further more, when that large table is in a task by it self, then the validation starts and it’s successful.

The DMS task is configured for both full-load and CDC (Change Data Capture). Here are the validation settings I'm using:

"ValidationSettings": {
      "ValidationPartialLobSize": 0,
      "PartitionSize": 10000,
      "RecordFailureDelayLimitInMinutes": 0,
      "SkipLobColumns": false,
      "FailureMaxCount": 10000,
      "HandleCollationDiff": false,
      "ValidationQueryCdcDelaySeconds": 0,
      "ValidationMode": "ROW_LEVEL",
      "TableFailureMaxCount": 1000,
      "RecordFailureDelayInMinutes": 5,
      "MaxKeyColumnSize": 8096,
      "EnableValidation": true,
      "ThreadCount": 5,
      "RecordSuspendDelayInMinutes": 30,
      "ValidationOnly": false
}

Given these settings, can someone suggest what might be causing the validation to not initiate for larger or numerous tables? Also, are there any recommended adjustments to the validation settings that might help resolve this issue? I tried playing with some of these settings like ParitionSize, FailureMaxCount, TableFiailureMaxCount, but nothing made any difference.

yatmee
asked 6 months ago108 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions