"kafka.bootstrap_servers" is not a configurable parameter for Amazon OpenSearch Ingestion pipelines

Question

I am trying to integrate confluent kafka topic with AWS OpenSearch via ingestion pipeline.

As per the official documentation of AWS OpenSearch, we can now use Kafka Plugin to integrate Kafka with Opensearch.

https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/kafka/

But I am getting an error when I am trying to configure an ingestion pipeline where the source is confluent kafka and destination(sink) is opensearch.

I am using the following config:

`version: '2'
kafka-pipeline:
  source:
    kafka:
      bootstrap_servers:
        - '127.x.x.x:xxxx'
      topics:
        - name: Topic1
          group_id: groupID1
  sink:
    - opensearch:
        hosts:
          - >-
https://xxxx.us-east-1.es.amazonaws.com
        aws:
          sts_role_arn: 'arn:aws:iam::xxxxx:role/xxxxx'
          region: us-east-1
        index_type: xxxxx`

But I am getting the following error upon config validation:

![error message](/media/postImages/original/IMTRyX_MawSYKG1FiOfRwGxg)

Answer

I have checked internally and got a confirmation that as of now OpenSearch ingestion only supports Amazon MSK. OpenSearch Ingestion compatibility with ‘Confluent Kafka’ is currently under development and is listed as one of the upcoming features. You can refer the below documentation which provides more information on ‘Using an OpenSearch Ingestion pipeline with Amazon Managed Streaming for Apache Kafka’

[+] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-msk.html

This explains the errors you are facing. If in case you need more details or technical support, feel free to reach out to AWS Premium Support with your queries.

"kafka.bootstrap_servers" is not a configurable parameter for Amazon OpenSearch Ingestion pipelines

Relevant content