Skip to content

Using Firehose transformation with Lambda, can one input record be mapped to multiple output records

0

Hello

I'm using firehose to get inputs from kinesis streams and to write to S3 tables, my question is, Can one input record be mapped to multiple output records?

Is this answer still valid https://repost.aws/en/questions/QUtAJymIMvSxORGmZPhIF26Q/can-i-split-a-batch-list-of-records-to-individual-records-while-using-firehose-lambda-transformations

1 Answer
2

The answer you referenced is still valid. When using Kinesis Firehose with Lambda transformations, it is not possible to map one input record to multiple output records. Kinesis Firehose expects to receive exactly one output record for each input record when using Lambda transformations.

The Lambda function used for transformation in Firehose can modify the content of the record, but it cannot create new records or split a single record into multiple records. This limitation is inherent to how Firehose processes data with Lambda transformations.

If you need to split a single record containing multiple items into separate records, you would need to perform this splitting operation before the data reaches Firehose. For example, you could do this in the source application that's sending data to Kinesis Streams, or you could use a separate Lambda function triggered by Kinesis Streams to split the records before they are sent to Firehose.

It's important to note that while you can't create multiple records from one during the Firehose Lambda transformation, you can still perform other types of transformations on the data. This includes modifying the content, enriching the data with additional information, or even dropping records based on certain conditions. However, the one-to-one relationship between input and output records must be maintained in the Lambda transformation step of Firehose.
Sources
Can I split a batch/list of records to individual records while using Firehose Lambda Transformations | AWS re:Post

answered a year ago
EXPERT
reviewed a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.