DMS Validation not running under some circumstances

1

I'm currently facing an issue with an AWS DMS task that involves validating data. My source database is Postgres, and the target for the data is an S3 bucket. Although I have enabled validation for the task, it's failing to run under certain conditions.

When my DMS task includes a large number of tables (around a dozen or more), the validation status remains stuck at "Pending validation". The logs provide minimal information, indicating an issue with the Athena manager not being able to locate the tables, stating they don't exist.

For example, the logs show messages like "S3 Target Validation Reached. Implementation In Progress," followed by "Athena Manager has no stored table name for table with iId 7," which suggests that it's unable to fetch the table name for the validation query.

Interestingly, when the task includes smaller tables, bellow 100k rows, the validation completes successfully. Issues arise when the task includes a large table, over 1-2 million rows. This problem occurs even with two tables, one small with 8k rows, and one big, with 5 million rows. Validation completes successfully for the small table, but won’t even start for the bigger table. When that table is on it’s own in a dms task, the validation completes .

Further more, when that large table is in a task by it self, then the validation starts and it’s successful.

The DMS task is configured for both full-load and CDC (Change Data Capture). Here are the validation settings I'm using:

"ValidationSettings": {
      "ValidationPartialLobSize": 0,
      "PartitionSize": 10000,
      "RecordFailureDelayLimitInMinutes": 0,
      "SkipLobColumns": false,
      "FailureMaxCount": 10000,
      "HandleCollationDiff": false,
      "ValidationQueryCdcDelaySeconds": 0,
      "ValidationMode": "ROW_LEVEL",
      "TableFailureMaxCount": 1000,
      "RecordFailureDelayInMinutes": 5,
      "MaxKeyColumnSize": 8096,
      "EnableValidation": true,
      "ThreadCount": 5,
      "RecordSuspendDelayInMinutes": 30,
      "ValidationOnly": false
}

Given these settings, can someone suggest what might be causing the validation to not initiate for larger or numerous tables? Also, are there any recommended adjustments to the validation settings that might help resolve this issue? I tried playing with some of these settings like ParitionSize, FailureMaxCount, TableFiailureMaxCount, but nothing made any difference.

yatmee
질문됨 6달 전112회 조회
답변 없음

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠