Fanning out DynamoDB stream into a kinesis stream using a lambda trigger?

0

Hi, I'm trying to implement a system which essentially provides in-order item-level information about DDB updates to multiple consumers. I'm using DDB streams as Kinesis streams from DDB don't maintain order and deduping (source). However, DDB streams can only have a max. of 2 concurrent consumers before being throttled.

So what I'm trying to do is have the DDB stream trigger a lambda, which serializes the com.amazonaws.services.lambda.runtime.events.DyanmodbEvent.DynamodbStreamRecord, and passes it into a kinesis stream. A few questions:

  1. What's the best method of serializing the DynamodbStreamRecord? I've seen some examples using KCL and the RecordAdapter, but that operates on the com.amazonaws.services.dynamodbv2.model.Record object, and not the lambda event objects DynamodbStreamRecord and StreamRecord.
  2. When I'm writing code to send data into a kinesis stream, the putRecord API in the kinesis client requires a partition key parameter. If I'm aiming to maintain the same order on an item level in the kinesis stream, should the partition key that I supply for the putRecord call just be the partition key of the relevant item? Thanks!
已提問 1 年前檢視次數 734 次
1 個回答
2
已接受的答案

Hi,

may not be a direct answer, but consider whether you could use EventBridge Pipes with DynamoStream as input instead: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-dynamodb.html.

From there, you could have the following targets as choice: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html.

I mention this option because I have experience that having single lambda as "manager" of the dynamo stream, will result in potential throttles and costs depending on the data traffic.

Hope it inspires you ;)

profile picture
專家
已回答 1 年前
profile picture
專家
已審閱 1 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南