Can I split a batch/list of records to individual records while using Firehose Lambda Transformations

0

Hi,

We have a scenario where we want to send a JSON object that contains a nested list of records into Kinesis Firehose from an IoT Core rule, and have this list split into multiple records, with each record then having some enrichment applied (new attribute added) before Firehose batches and writes these individual records to S3 (Parquet).

Is the splitting something that we are able to do via a Firehose Lambda Transformation function, or would we need to split the list before Firehose, and then pass each record in? The issue I think we'd hit with the former is the fact that the incoming payload would have a single recordId, and Firehose won't like us passing the same ID back for multiple records.

Example incoming payload from IoT Core:

{ "records": [ { "id": "1", "name": "Name 1" }, { "id": "2", "name": "Name 2" } ] }

So in the ideal scenario, the two items in the records array would be enriched separately and returned to Firehose as multiple records.

1 Risposta
0
Risposta accettata

No. That is not possible. Kinesis Firehose expects to receive one output record for each input record. You can't create new records in the function.

profile pictureAWS
ESPERTO
Uri
con risposta 4 mesi fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande