Join Streaming Data in Amazon Kinesis Firehose with Data in S3 via Athena using a Data transformation Lambda function

0

Can a Lambda data transformation function attached to a Kinesis Firehose perform a join between the streaming data in that Kinesis Firehose and data in S3 through Athena, and pass the results to Kinesis Analytics for further processing? Is this functionality supported?

feita há 3 anos769 visualizações
1 Resposta
0
Resposta aceita

Theoretically yes.

However, the Lambda synchronous invocation mode has a payload size limit of 6 MB for both the request and the response. Make sure that your buffering size for sending the request to the function is less than or equal to 6 MB. Also ensure that the response that your function returns doesn't exceed 6 MB.

Kinesis Data Firehose supports Lambda executions limited to 5 minutes per invocation. If your Lambda function exceeds 5 minutes you get the following error: Firehose encountered timeout errors when calling AWS Lambda. The maximum supported function timeout is 5 minutes.

The preferred method is to perform a lookup instead of query. Consider querying a DynamoDB table.

Also, consider pre-processing records using Lambda functions attached to your Kinesis Data Analytics application and performing reference data lookups in S3. Read this - > https://docs.aws.amazon.com/kinesisanalytics/latest/dev/lambda-preprocessing.html and https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html

AWS
ESPECIALISTA
respondido há 3 anos

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas