OpenSearch Data Migration Ingestion Pipeline Error: Missing required property 'ElasticsearchVersionInfo.buildFlavor'

Question

Hello support,
I am trying to setup an OpenSearch Data Migration Migration pipeline from a Managed Cluster Domain Indices to a Serverless Collection (within the same account and region). I have followed the documentation on how to proceed and setup the configuration.

After pipelines initiates I see the following exception in CloudWatch logs: 
```
[opensearch-migration-pipeline-sink-worker-2-thread-1] ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [opensearch-migration-pipeline] process worker encountered a fatal exception, cannot proceed further
java.lang.RuntimeException: Unable to call info API using the elasticsearch client
caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'ElasticsearchVersionInfo.buildFlavor'
```

Note: Managed Cluster Domain is running latest Engine version 2.15

As a result no index gets migrated. Am I missing something in my configuration? any other suggestion on how to migrate indices from OS Domain to OS Serverless Collection?
Thank you very much in advance!
-Ransel

Almas-AWS · Accepted Answer

Hi

This can happen if you are running source Amazon OpenSearch domain with compatibility mode enabled and `distribution_version` is not specified in the pipeline configuration YAML template.

Can you try adding it to the source as below:

```
version: "2"
opensearch-migration-pipeline:
  source:
    opensearch:
      acknowledgments: true
      # Provide an OpenSearch or Elasticsearch cluster endpoint.
      hosts: [ "<>" ]
      # Uncomment the distribution_version below if you are using an Amazon OpenSearch domain with compatibility mode enabled
      distribution_version: "opensearch"
      .....
      .....
```

Ransel · Answer

Hi,
Thank you so much for your promptly and accurate response. This solution solved the issue.

OpenSearch Data Migration Ingestion Pipeline Error: Missing required property 'ElasticsearchVersionInfo.buildFlavor'

Add your answer

Relevant content