Can I split a batch/list of records to individual records while using Firehose Lambda Transformations

0

Hi,

We have a scenario where we want to send a JSON object that contains a nested list of records into Kinesis Firehose from an IoT Core rule, and have this list split into multiple records, with each record then having some enrichment applied (new attribute added) before Firehose batches and writes these individual records to S3 (Parquet).

Is the splitting something that we are able to do via a Firehose Lambda Transformation function, or would we need to split the list before Firehose, and then pass each record in? The issue I think we'd hit with the former is the fact that the incoming payload would have a single recordId, and Firehose won't like us passing the same ID back for multiple records.

Example incoming payload from IoT Core:

{ "records": [ { "id": "1", "name": "Name 1" }, { "id": "2", "name": "Name 2" } ] }

So in the ideal scenario, the two items in the records array would be enriched separately and returned to Firehose as multiple records.

cgddrd
preguntada hace 4 meses892 visualizaciones
1 Respuesta
0
Respuesta aceptada

No. That is not possible. Kinesis Firehose expects to receive one output record for each input record. You can't create new records in the function.

profile pictureAWS
EXPERTO
Uri
respondido hace 4 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas