Multiple tables in one Data Pipeline?

0

I am using Data Pipelines to replicate data from RDS to Redshift with one pipeline per table they are replicating.

Is it possible to consolidate the pipelines into a single pipeline for multiple tables? And if so, is there an example JSON config that would make this work?

1개 답변
0
수락된 답변

Yes, this is possible. Below I've included a demo JSON code for that pipeline.

Note that the JSON isn't complete as there are more attributes required for MySql and Redshift - I've included links to the full attribute requirements as well.

MySqlDataNode Requirements

RedshiftDataNode Requirements

{
  "objects": [
    {
      "output": {
        "ref": "RedshiftDataNodeId_BWynX"
      },
      "input": {
        "ref": "MySqlDataNodeId_iOnTY"
      },
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultCopyActivity2",
      "id": "CopyActivityId_WMYod",
      "type": "CopyActivity"
    },
    {
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultMySqlDataNode3",
      "id": "MySqlDataNodeId_9uoCY",
      "type": "MySqlDataNode",
      "table": ""
    },
    {
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultRedshiftDataNode1",
      "id": "RedshiftDataNodeId_mJUEj",
      "type": "RedshiftDataNode",
      "tableName": ""
    },
    {
      "output": {
        "ref": "RedshiftDataNodeId_yz06G"
      },
      "input": {
        "ref": "MySqlDataNodeId_9uoCY"
      },
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultCopyActivity3",
      "id": "CopyActivityId_nm5LG",
      "type": "CopyActivity"
    },
    {
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultMySqlDataNode1",
      "id": "MySqlDataNodeId_6Xe5h",
      "type": "MySqlDataNode",
      "table": ""
    },
    {
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultRedshiftDataNode3",
      "id": "RedshiftDataNodeId_yz06G",
      "type": "RedshiftDataNode",
      "tableName": ""
    },
    {
      "failureAndRerunMode": "CASCADE",
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "resourceRole": "DataPipelineDefaultResourceRole",
      "role": "DataPipelineDefaultRole",
      "pipelineLogUri": "s3://athena-ab3/",
      "scheduleType": "cron",
      "name": "Default",
      "id": "Default"
    },
    {
      "occurrences": "1",
      "period": "1 days",
      "name": "RunOnce",
      "id": "DefaultSchedule",
      "type": "Schedule",
      "startAt": "FIRST_ACTIVATION_DATE_TIME"
    },
    {
      "output": {
        "ref": "RedshiftDataNodeId_mJUEj"
      },
      "input": {
        "ref": "MySqlDataNodeId_6Xe5h"
      },
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultCopyActivity1",
      "id": "CopyActivityId_G0Lps",
      "type": "CopyActivity"
    },
    {
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultRedshiftDataNode2",
      "id": "RedshiftDataNodeId_BWynX",
      "type": "RedshiftDataNode",
      "tableName": ""
    },
    {
      "schedule": {
        "ref": "DefaultSchedule"
      },
      "name": "DefaultMySqlDataNode2",
      "id": "MySqlDataNodeId_iOnTY",
      "type": "MySqlDataNode",
      "table": ""
    }
  ],
  "parameters": []
}
AWS
답변함 3년 전
profile picture
전문가
검토됨 한 달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠