Fanning out DynamoDB stream into a kinesis stream using a lambda trigger?

0

Hi, I'm trying to implement a system which essentially provides in-order item-level information about DDB updates to multiple consumers. I'm using DDB streams as Kinesis streams from DDB don't maintain order and deduping (source). However, DDB streams can only have a max. of 2 concurrent consumers before being throttled.

So what I'm trying to do is have the DDB stream trigger a lambda, which serializes the com.amazonaws.services.lambda.runtime.events.DyanmodbEvent.DynamodbStreamRecord, and passes it into a kinesis stream. A few questions:

  1. What's the best method of serializing the DynamodbStreamRecord? I've seen some examples using KCL and the RecordAdapter, but that operates on the com.amazonaws.services.dynamodbv2.model.Record object, and not the lambda event objects DynamodbStreamRecord and StreamRecord.
  2. When I'm writing code to send data into a kinesis stream, the putRecord API in the kinesis client requires a partition key parameter. If I'm aiming to maintain the same order on an item level in the kinesis stream, should the partition key that I supply for the putRecord call just be the partition key of the relevant item? Thanks!
1回答
2
承認された回答

Hi,

may not be a direct answer, but consider whether you could use EventBridge Pipes with DynamoStream as input instead: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-dynamodb.html.

From there, you could have the following targets as choice: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html.

I mention this option because I have experience that having single lambda as "manager" of the dynamo stream, will result in potential throttles and costs depending on the data traffic.

Hope it inspires you ;)

profile picture
エキスパート
回答済み 1年前
profile picture
エキスパート
レビュー済み 1ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ