DynamoDB to OpenSearch pipeline issue

Question

Hi all,
Following this tutorial https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-ddb.html.
I have shared opensearch pipeline configuration for sinking all dynamoDB updates to Open search domain. I enabled cloud watch logs and it is not giving any error and i can see the opensaerch sink is successfuly initialised.  However when I create item in dynamoDB table, its not reaching opensearch and no logs are produced for this create item stream in dynamoDB. Is there anything that we miss in out setup?
   
```
 version: "2"
dynamodb-pipeline:
  source:
    dynamodb:
      acknowledgments: true
      tables:
        - table_arn: "arn:aws:dynamodb:ap-south-1:{ACC_ID}:table/Application"
          stream:
            start_position: "LATEST"
      aws:
         sts_role_arn: "{PIPELINE_ROLE}"
         region: "ap-south-1"
  sink:
    - opensearch:
        # REQUIRED: Provide an AWS OpenSearch endpoint
        hosts: [ "https://search-abcd.ap-south-1.es.amazonaws.com" ]
        index: "application"
        index_type: custom
        document_id: "${getMetadata(\"primary_key\")}"
        action: "${getMetadata(\"opensearch_action\")}"
        document_version: "${getMetadata(\"document_version\")}"
        document_version_type: "external"
        aws:
             sts_role_arn: "{PIPELINE_ROLE}"
             region: "ap-south-1"
   
```

Aman Dhingra · Answer

1. Do you see any datapoints in ReturnedRecordsCount CW metrics on DynamoDB Stream? Non-zero datapoints here would indicate that the DynamoDB Stream is being consumed. If no datapoints, then OpenSearch role is unable to call GetRecords on the DynamoDB Stream, so permissions need fixing. If yes, then DDBStream permissions are fine, but need to investigate on the OpenSearch side.

1. recordsWriteFailed.count CW metric on the OpenSearch pipeline will show datapoints as well, if the pipeline attempted to write to OpenSearch and failed. If no datapoints (and datapoints in (1)) then pipeline is able to write to OS. If no datapoints, then investigate the "data access policies" of the OpenSearch domain/cluster. The pipeline role must be allowed permissions to write there.

1. If datapoints in (1) and no datapoints in (2), then must ensure DDBStream records are being generated. For a DDBStream, if an item is overwritten with the same exact values, then no Stream records are generated.

These are some of the common things to check for while troubleshooting the DDB<>OS pipeline. It would help if you try to write records to OS directly using the PipelineRole to ensure data can be written successfully. If this works, then try re-creating the pipeline. CW log group with name like /aws/vendedlogs///audit-logs should have logs on the pipeline consuming the DDBStream shards. If not, then logging permissions need fixing.

DynamoDB to OpenSearch pipeline issue

Relevant content