Data ingestion in Opensearch Serverless

Question

I want to ingest logs from onpremise syslog server to opensearch serverless service. I am transferring now the log files to S3 and using lambda, I am ingesting data to to opensearch serverless collection. Can I continue to do that or is there any other efficient way to do it?

Accepted Answer

You can use Kinesis Firehose to automate that with some buffering ( it will help to call in batches to OpenSearch, that is automatically done by Firehose).

Are you doing this with lambda? I mean, your approach is valid, but maybe just using Kinesis Firehose from your On-premises Server is enough.

You can read about this approach here.

https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html

Bests

Answer

HI,

The Amazon OpenSearch service is compatible [with multiple integration tools](https://aws.amazon.com/opensearch-service/features/integration/) for ingesting data from different data sources , but choosing the right ingestion mechanism depends on your use case requirements such as data latency and data type.

Thus, If you need to ingest logs in real time, a common solution is to use the [Kinesis Agent](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html) to send data to Kinesis Data Firehose and [configure your OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/integrations.html#integrations-fh) as the delivery destination. Kinesis Data Firehose is a fully managed auto scalable service, so requires no ongoing administration, and allows transforming data via AWS Lambda function.

Data ingestion in Opensearch Serverless

Relevant content