Can I split a batch/list of records to individual records while using Firehose Lambda Transformations

0

Hi,

We have a scenario where we want to send a JSON object that contains a nested list of records into Kinesis Firehose from an IoT Core rule, and have this list split into multiple records, with each record then having some enrichment applied (new attribute added) before Firehose batches and writes these individual records to S3 (Parquet).

Is the splitting something that we are able to do via a Firehose Lambda Transformation function, or would we need to split the list before Firehose, and then pass each record in? The issue I think we'd hit with the former is the fact that the incoming payload would have a single recordId, and Firehose won't like us passing the same ID back for multiple records.

Example incoming payload from IoT Core:

{ "records": [ { "id": "1", "name": "Name 1" }, { "id": "2", "name": "Name 2" } ] }

So in the ideal scenario, the two items in the records array would be enriched separately and returned to Firehose as multiple records.

1 Antwort
0
Akzeptierte Antwort

No. That is not possible. Kinesis Firehose expects to receive one output record for each input record. You can't create new records in the function.

profile pictureAWS
EXPERTE
Uri
beantwortet vor 4 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen