Ingesting data from external sources like Git, Slack, Zoom, Instagram, 3rd party systems

Question

## Problem
I want to know, understand and correct my knowledge, approach on, Setting up an Data Ingestion pipeline, which collects "events" or "data" from any possible external application sources (applications of 3rd party)

The rate of ingestion can be about 5000 (5K) events per day (normal) on peak it can go slightly more 20K

### Approach I been thinking about
I am planning to setup AWS Lambda endpoint to which external systems can post(HTTP POST) the data, which then can load into OpenSearch to form a Data Lake

The ingestion pipeline operates between Lambda and OpenSearch, to perform
- Parsing of data
- Fetch more data if needed, by making API calls
- Process, transform, enrich
- Post to OpenSearch Indices as per indices

I have been googling and exploring on AWS but so far can't find any thing which can validate above. Hence request you experts to comment, suggest and direct me to a practical solution

Answer

Hi, 
Based on the ingestion rates, I think that the architecture you envisioned should work well.

It is a variant of [this pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.html) , the difference is that you use [the Lambda to load Opensearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/integrations.html) and not to trigger a search.

The other pattern you could consider is to use [API Gateway as a proxy to Amazon Kinesis Firehose](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-as-kinesis-proxy-export-swagger-with-extensions.html) (please, note the tutorial is for Kinesis Streams, and was only meant as a source of inspiration), and then use [Firehose to transform the data](https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html) (it will still uses Lambda functions for the transformations) and then deliver the data directly to OpenSearch.

Some information additional information on this can be

Ingesting data from external sources like Git, Slack, Zoom, Instagram, 3rd party systems

Problem

Approach I been thinking about

Contenuto pertinente