Can I split a batch/list of records to individual records while using Firehose Lambda Transformations

0

Hi,

We have a scenario where we want to send a JSON object that contains a nested list of records into Kinesis Firehose from an IoT Core rule, and have this list split into multiple records, with each record then having some enrichment applied (new attribute added) before Firehose batches and writes these individual records to S3 (Parquet).

Is the splitting something that we are able to do via a Firehose Lambda Transformation function, or would we need to split the list before Firehose, and then pass each record in? The issue I think we'd hit with the former is the fact that the incoming payload would have a single recordId, and Firehose won't like us passing the same ID back for multiple records.

Example incoming payload from IoT Core:

{ "records": [ { "id": "1", "name": "Name 1" }, { "id": "2", "name": "Name 2" } ] }

So in the ideal scenario, the two items in the records array would be enriched separately and returned to Firehose as multiple records.

1개 답변
0
수락된 답변

No. That is not possible. Kinesis Firehose expects to receive one output record for each input record. You can't create new records in the function.

profile pictureAWS
전문가
Uri
답변함 4달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠