AWS DMS Postgres to OpenSearch LOB handling

1

Source: postgres Target: OpenSearch

I have a text column called description in one of my postgres tables. Per the documentation, this data type is mapped to a NCLOB. Since OpenSearch does not not offer LOB support, my description is missing in my OpenSearch documents.

I tried using the mapping rule bellow, but does not seem to be doing anything

{
    "rule-type": "transformation",
    "rule-id": "3",
    "rule-name": "3",
    "rule-target": "column",
    "object-locator": {
        "schema-name": "public",
        "table-name": "jobs",
        "column-name": "description"
    },
    "rule-action": "change-data-type",
    "data-type": {
        "type": "string",
        "length": 500
    }
}

When i check the logs i see the following

Column 'description' is unsupported in table def 'public.jobs' since the LOB support is disabled 

However, i do have LOB enabled under task settings:

"TargetMetadata": {
        "ParallelApplyBufferSize": 0,
        "ParallelApplyQueuesPerThread": 0,
        "ParallelApplyThreads": 0,
        "TargetSchema": "",
        "InlineLobMaxSize": 0,
        "ParallelLoadQueuesPerThread": 0,
        "SupportLobs": true,
        "LobChunkSize": 10,
        "TaskRecoveryTableEnabled": false,
        "ParallelLoadThreads": 0,
        "BatchApplyEnabled": false,
        "FullLobMode": true,
        "LimitedSizeLobMode": false,
        "LoadMaxFileSize": 0,
        "ParallelLoadBufferSize": 0
    },

Is that transformation rule supposed to work? Or will any LOB column be skipped because OpenSearch does not have LOB support? Any way to make this work?

Thanks!

1 個回答
0

Transformation rules like - "rule-action": "change-data-type", work on DMS data types. "type": "string" , "length": 500 will be the DMS data type here which will be converted to the target engine data type. Since DMS does not support string as a data type for Opensearch target, this attribute is not getting converted. Please refer : https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Elasticsearch.html#CHAP_Target.Elasticsearch.DataTypes AWS DMS data type - Boolean,Date,Time, Timestamp,INT4,Real4,UINT4 only supported DMS data types. No string data type allowed for Opensearch as a target

AWS
已回答 2 年前
  • How does it handle varchars? in the same table i have a name column which is a varchar(255) and that get's mapped to a string type in OpenSearch. Is there no way to migrate a text type from postgres to OpenSearch?

  • Yeah, it's confusing how DMS handles varchar even it is not mentioned in the documentation as supported source data type! I'm stuck in the same problem, did you find any solution?

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南