Using opensearch pipelines to retrieve data from http endpoint that requires authentication

Question

I want to ingest data using an opensearch pipeline (dataprepper).

My endpoint is a CRM system with an API,

The workflow looks like this:

1. authenticate with server, receive token
2. use creds stored in secrets manager + token to make a POST request to an API endpoint to get a list of results
3. other API requests to enrich the data

Does data prepper support fetching data from an authenticated http endpoint, as opposed to waiting to receive it, like a web hook.

If so anyone got any examples of templates with this sort of thing?

Yev · Answer

Hello,

According to Data Prepper's [`http` source documentation](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/http/), only HTTP Basic Auth is supported "out of the box". So, you'll have to implement an authentication plugin for the Data Prepper.

Judging by your list, I might recommend using a set of Lambda functions to retrieve data sets and save them into an S3 bucket. Then, you could load the data into Open Search using Data Prepper's pipeline with an `s3` source.

However, be aware, that Data Prepper is not a part of the Open Search deployment package, but a separate Docker container. Thus,  you will need a place to run it (EC2, ECS etc).

Using opensearch pipelines to retrieve data from http endpoint that requires authentication

Add your answer

Relevant content