如何使用 AWS CLI 和 jq 建立 AWS DMS 任務設定的備份？

簡短說明

使用 AWS Command Line Interface (AWS CLI) 和 jq 命令列 JSON 處理器來建立 AWS DMS 複寫任務設定的備份。

解決方法

**注意：**如果您在執行 AWS CLI 命令時收到錯誤，請參閱對 AWS CLI 錯誤進行疑難排解。此外，請確定您使用的是最新的 AWS CLI 版本。

先決條件

1. 安裝 AWS CLI。
2. 從 GitHub 上的 ./jq 儲存庫下載 jq 工具。

取得複寫任務設定

若要取得 AWS DMS 複寫任務設定，請使用 AWS CLI 執行 describe-replication-tasks 命令。您必須使用 --filters 旗標。使用 filters 旗標，您只會傳回符合下列其中一個項目的任務：

• 複寫任務 ID
• 複寫任務 ARN

執行此查詢以取得具有複寫任務 ID 的 AWS DMS 任務的複寫任務設定。將 dms_task_id 取代為您的 AWS DMS 複寫任務 ID。

aws dms describe-replication-tasks --filter Name=replication-task-id,Values=dms_task_id --output json

執行此查詢以取得具有複寫任務 ARN 的 AWS DMS 任務的複寫任務設定。將 dms_task_arn 取代為您的 AWS DMS 複寫任務 ARN。

aws dms describe-replication-tasks --filter Name=replication-task-id,Values=dms_task_arn --output json

使用 jq 篩選並轉換 JSON 輸出

根據您要執行的任務類型，篩選並轉換前一個步驟的 JSON 輸出。您必須篩選和轉換此 JSON，來使其相容以進行稍後還原。

使用 jq 命令列 JSON 處理器來修改輸出 JSON：

**全負載任務 **

移除下列資料：

• ReplicationTaskArn
• ReplicationTaskStartDate
• ReplicationTaskStats
• Status
• StopReason
• ReplicationTaskCreationDate
• CloudWatchLogGroup
• CloudWatchLogStream
• RecoveryCheckpoint

**CDC 任務 **

• 新增鍵值 CdcStartPosition。然後，將值設定為 RecoveryCheckpoint。

移除下列資料：

• ReplicationTaskArn
• ReplicationTaskStartDate
• ReplicationTaskStats
• Status
• StopReason
• ReplicationTaskCreationDate
• CloudWatchLogGroup
• CloudWatchLogStream
• RecoveryCheckpoint

**全負載和 CDC 任務 **

移除下列資料：

• ReplicationTaskArn
• ReplicationTaskStartDate
• ReplicationTaskStats
• Status
• StopReason
• ReplicationTaskCreationDate
• CloudWatchLogGroup
• CloudWatchLogStream
• RecoveryCheckpoint

然後，執行下列其中一個命令。這些命令將輸出 JSON 導入 jq，然後篩選並轉換複寫任務資料。

使用 AWS DMS 複寫任務 ID

aws dms describe-replication-tasks --filter Name=replication-task-id,Values=dms_task_id --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([    ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]])
    | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring'

使用 AWS DMS 複寫任務 ARN

aws dms describe-replication-tasks --filter Name=replication-task-arn,Values=dms-task-arn --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([    ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]])
    | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring'

將輸出儲存至 JSON 檔案

若要將產生的輸出另存為備份，請將其重導向至 JSON 檔案。例如，此命令會儲存具有名稱為 task_backup.json 的 JSON 檔案的任務 ID dms_original_task 的 AWS DMS 任務的任務設定：

aws dms describe-replication-tasks --filter Name=replication-task-id,Values=original-dms-task --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([ ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]]) | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring' > task_backup.json

輸出 JSON 檔案會顯示類似於下列內容的輸出：

{  "ReplicationTaskIdentifier": "original-dms-task",
  "SourceEndpointArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:endpoint:QKMQN2TCULCPYIK3CMGF2CRDC5DIWZ433BCKHZY",
  "TargetEndpointArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:endpoint:6LEFAZCDPQD2HWHJIXJYUU4PGMTKA2AKKFN4KCA",
  "ReplicationInstanceArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:rep:C4CAB7ANKQU2AU7WFH4BYCJSLPS3YYUGCTHL3UA",
  "MigrationType": "full-load-and-cdc",
  "TableMappings": "{\"rules\":[{\"rule-type\":\"selection\",\"rule-id\":\"293111713\",\"rule-name\":\"293111713\",\"object-locator\":{\"schema-name\":\"%\",\"table-name\":\"%\"},\"rule-action\":\"include\",\"filters\":[]}]}",
  "ReplicationTaskSettings": "{\"Logging\":{\"EnableLogging\":true,\"LogComponents\":[{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TRANSFORMATION\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SOURCE_UNLOAD\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"IO\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TARGET_LOAD\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"PERFORMANCE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SOURCE_CAPTURE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SORTER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"REST_SERVER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"VALIDATOR_EXT\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TARGET_APPLY\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TASK_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TABLES_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"METADATA_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"FILE_FACTORY\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"COMMON\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"ADDONS\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"DATA_STRUCTURE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"COMMUNICATION\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"FILE_TRANSFER\"}]},\"StreamBufferSettings\":{\"StreamBufferCount\":3,\"CtrlStreamBufferSizeInMB\":5,\"StreamBufferSizeInMB\":8},\"ErrorBehavior\":{\"FailOnNoTablesCaptured\":true,\"ApplyErrorUpdatePolicy\":\"LOG_ERROR\",\"FailOnTransactionConsistencyBreached\":false,\"RecoverableErrorThrottlingMax\":1800,\"DataErrorEscalationPolicy\":\"SUSPEND_TABLE\",\"ApplyErrorEscalationCount\":0,\"RecoverableErrorStopRetryAfterThrottlingMax\":true,\"RecoverableErrorThrottling\":true,\"ApplyErrorFailOnTruncationDdl\":false,\"DataTruncationErrorPolicy\":\"LOG_ERROR\",\"ApplyErrorInsertPolicy\":\"LOG_ERROR\",\"EventErrorPolicy\":\"IGNORE\",\"ApplyErrorEscalationPolicy\":\"LOG_ERROR\",\"RecoverableErrorCount\":-1,\"DataErrorEscalationCount\":0,\"TableErrorEscalationPolicy\":\"STOP_TASK\",\"RecoverableErrorInterval\":5,\"ApplyErrorDeletePolicy\":\"IGNORE_RECORD\",\"TableErrorEscalationCount\":0,\"FullLoadIgnoreConflicts\":true,\"DataErrorPolicy\":\"LOG_ERROR\",\"TableErrorPolicy\":\"SUSPEND_TABLE\"},\"ValidationSettings\":{\"ValidationPartialLobSize\":0,\"PartitionSize\":10000,\"RecordFailureDelayLimitInMinutes\":0,\"SkipLobColumns\":false,\"FailureMaxCount\":10000,\"HandleCollationDiff\":false,\"ValidationQueryCdcDelaySeconds\":0,\"ValidationMode\":\"ROW_LEVEL\",\"TableFailureMaxCount\":1000,\"RecordFailureDelayInMinutes\":5,\"MaxKeyColumnSize\":8096,\"EnableValidation\":true,\"ThreadCount\":5,\"RecordSuspendDelayInMinutes\":30,\"ValidationOnly\":false},\"TTSettings\":{\"TTS3Settings\":null,\"TTRecordSettings\":null,\"EnableTT\":false},\"FullLoadSettings\":{\"CommitRate\":10000,\"StopTaskCachedChangesApplied\":false,\"StopTaskCachedChangesNotApplied\":false,\"MaxFullLoadSubTasks\":8,\"TransactionConsistencyTimeout\":600,\"CreatePkAfterFullLoad\":false,\"TargetTablePrepMode\":\"DROP_AND_CREATE\"},\"TargetMetadata\":{\"ParallelApplyBufferSize\":0,\"ParallelApplyQueuesPerThread\":0,\"ParallelApplyThreads\":0,\"TargetSchema\":\"\",\"InlineLobMaxSize\":0,\"ParallelLoadQueuesPerThread\":0,\"SupportLobs\":true,\"LobChunkSize\":0,\"TaskRecoveryTableEnabled\":false,\"ParallelLoadThreads\":0,\"LobMaxSize\":32,\"BatchApplyEnabled\":false,\"FullLobMode\":false,\"LimitedSizeLobMode\":true,\"LoadMaxFileSize\":0,\"ParallelLoadBufferSize\":0},\"BeforeImageSettings\":null,\"ControlTablesSettings\":{\"historyTimeslotInMinutes\":5,\"HistoryTimeslotInMinutes\":5,\"StatusTableEnabled\":false,\"SuspendedTablesTableEnabled\":false,\"HistoryTableEnabled\":false,\"ControlSchema\":\"\",\"FullLoadExceptionTableEnabled\":false},\"LoopbackPreventionSettings\":null,\"CharacterSetSettings\":null,\"FailTaskWhenCleanTaskResourceFailed\":false,\"ChangeProcessingTuning\":{\"StatementCacheSize\":50,\"CommitTimeout\":1,\"BatchApplyPreserveTransaction\":true,\"BatchApplyTimeoutMin\":1,\"BatchSplitSize\":0,\"BatchApplyTimeoutMax\":30,\"MinTransactionSize\":1000,\"MemoryKeepTime\":60,\"BatchApplyMemoryLimit\":500,\"MemoryLimitTotal\":1024},\"ChangeProcessingDdlHandlingPolicy\":{\"HandleSourceTableDropped\":true,\"HandleSourceTableTruncated\":true,\"HandleSourceTableAltered\":true},\"PostProcessingRules\":null}",
  "CdcStartPosition": "checkpoint:V1#102#00000000/B803EBA0#0#0#*#0#277"
}

您可以使用 create-replication-task AWS CLI 命令來編輯和還原 JSON 檔案。

例如，此命令會使用 task_backup.json 檔案來建立具有 restored-dms-task 的任務 ID 的 AWS DMS 複寫任務：

aws dms create-replication-task --replication-task-identifier restored-dms-task --cli-input-json file://task_backup.json

注意： 如果您在命令中包含 replication-task-identifier，則您會覆寫 task_backup.json 檔案中的 replication-task-identifier 值。

相關資訊

指定 AWS Database Migration Service 任務的任務設定

使用 AWS DMS 建立持續複寫的任務