Join Streaming Data in Amazon Kinesis Firehose with Data in S3 via Athena using a Data transformation Lambda function

0

Can a Lambda data transformation function attached to a Kinesis Firehose perform a join between the streaming data in that Kinesis Firehose and data in S3 through Athena, and pass the results to Kinesis Analytics for further processing? Is this functionality supported?

質問済み 3年前769ビュー
1回答
0
承認された回答

Theoretically yes.

However, the Lambda synchronous invocation mode has a payload size limit of 6 MB for both the request and the response. Make sure that your buffering size for sending the request to the function is less than or equal to 6 MB. Also ensure that the response that your function returns doesn't exceed 6 MB.

Kinesis Data Firehose supports Lambda executions limited to 5 minutes per invocation. If your Lambda function exceeds 5 minutes you get the following error: Firehose encountered timeout errors when calling AWS Lambda. The maximum supported function timeout is 5 minutes.

The preferred method is to perform a lookup instead of query. Consider querying a DynamoDB table.

Also, consider pre-processing records using Lambda functions attached to your Kinesis Data Analytics application and performing reference data lookups in S3. Read this - > https://docs.aws.amazon.com/kinesisanalytics/latest/dev/lambda-preprocessing.html and https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html

AWS
エキスパート
回答済み 3年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ