Fanning out DynamoDB stream into a kinesis stream using a lambda trigger?

0

Hi, I'm trying to implement a system which essentially provides in-order item-level information about DDB updates to multiple consumers. I'm using DDB streams as Kinesis streams from DDB don't maintain order and deduping (source). However, DDB streams can only have a max. of 2 concurrent consumers before being throttled.

So what I'm trying to do is have the DDB stream trigger a lambda, which serializes the com.amazonaws.services.lambda.runtime.events.DyanmodbEvent.DynamodbStreamRecord, and passes it into a kinesis stream. A few questions:

  1. What's the best method of serializing the DynamodbStreamRecord? I've seen some examples using KCL and the RecordAdapter, but that operates on the com.amazonaws.services.dynamodbv2.model.Record object, and not the lambda event objects DynamodbStreamRecord and StreamRecord.
  2. When I'm writing code to send data into a kinesis stream, the putRecord API in the kinesis client requires a partition key parameter. If I'm aiming to maintain the same order on an item level in the kinesis stream, should the partition key that I supply for the putRecord call just be the partition key of the relevant item? Thanks!
1 Risposta
2
Risposta accettata

Hi,

may not be a direct answer, but consider whether you could use EventBridge Pipes with DynamoStream as input instead: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-dynamodb.html.

From there, you could have the following targets as choice: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html.

I mention this option because I have experience that having single lambda as "manager" of the dynamo stream, will result in potential throttles and costs depending on the data traffic.

Hope it inspires you ;)

profile picture
ESPERTO
con risposta un anno fa
profile picture
ESPERTO
verificato un mese fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande