Fanning out DynamoDB stream into a kinesis stream using a lambda trigger?

0

Hi, I'm trying to implement a system which essentially provides in-order item-level information about DDB updates to multiple consumers. I'm using DDB streams as Kinesis streams from DDB don't maintain order and deduping (source). However, DDB streams can only have a max. of 2 concurrent consumers before being throttled.

So what I'm trying to do is have the DDB stream trigger a lambda, which serializes the com.amazonaws.services.lambda.runtime.events.DyanmodbEvent.DynamodbStreamRecord, and passes it into a kinesis stream. A few questions:

  1. What's the best method of serializing the DynamodbStreamRecord? I've seen some examples using KCL and the RecordAdapter, but that operates on the com.amazonaws.services.dynamodbv2.model.Record object, and not the lambda event objects DynamodbStreamRecord and StreamRecord.
  2. When I'm writing code to send data into a kinesis stream, the putRecord API in the kinesis client requires a partition key parameter. If I'm aiming to maintain the same order on an item level in the kinesis stream, should the partition key that I supply for the putRecord call just be the partition key of the relevant item? Thanks!
1개 답변
2
수락된 답변

Hi,

may not be a direct answer, but consider whether you could use EventBridge Pipes with DynamoStream as input instead: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-dynamodb.html.

From there, you could have the following targets as choice: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html.

I mention this option because I have experience that having single lambda as "manager" of the dynamo stream, will result in potential throttles and costs depending on the data traffic.

Hope it inspires you ;)

profile picture
전문가
답변함 일 년 전
profile picture
전문가
검토됨 한 달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠