Join Streaming Data in Amazon Kinesis Firehose with Data in S3 via Athena using a Data transformation Lambda function

0

Can a Lambda data transformation function attached to a Kinesis Firehose perform a join between the streaming data in that Kinesis Firehose and data in S3 through Athena, and pass the results to Kinesis Analytics for further processing? Is this functionality supported?

gefragt vor 3 Jahren769 Aufrufe
1 Antwort
0
Akzeptierte Antwort

Theoretically yes.

However, the Lambda synchronous invocation mode has a payload size limit of 6 MB for both the request and the response. Make sure that your buffering size for sending the request to the function is less than or equal to 6 MB. Also ensure that the response that your function returns doesn't exceed 6 MB.

Kinesis Data Firehose supports Lambda executions limited to 5 minutes per invocation. If your Lambda function exceeds 5 minutes you get the following error: Firehose encountered timeout errors when calling AWS Lambda. The maximum supported function timeout is 5 minutes.

The preferred method is to perform a lookup instead of query. Consider querying a DynamoDB table.

Also, consider pre-processing records using Lambda functions attached to your Kinesis Data Analytics application and performing reference data lookups in S3. Read this - > https://docs.aws.amazon.com/kinesisanalytics/latest/dev/lambda-preprocessing.html and https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html

AWS
EXPERTE
beantwortet vor 3 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen