Can I split a batch/list of records to individual records while using Firehose Lambda Transformations

0

Hi,

We have a scenario where we want to send a JSON object that contains a nested list of records into Kinesis Firehose from an IoT Core rule, and have this list split into multiple records, with each record then having some enrichment applied (new attribute added) before Firehose batches and writes these individual records to S3 (Parquet).

Is the splitting something that we are able to do via a Firehose Lambda Transformation function, or would we need to split the list before Firehose, and then pass each record in? The issue I think we'd hit with the former is the fact that the incoming payload would have a single recordId, and Firehose won't like us passing the same ID back for multiple records.

Example incoming payload from IoT Core:

{ "records": [ { "id": "1", "name": "Name 1" }, { "id": "2", "name": "Name 2" } ] }

So in the ideal scenario, the two items in the records array would be enriched separately and returned to Firehose as multiple records.

1 Resposta
0
Resposta aceita

No. That is not possible. Kinesis Firehose expects to receive one output record for each input record. You can't create new records in the function.

profile pictureAWS
ESPECIALISTA
Uri
respondido há 4 meses

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas