Dynamo DB Kinesis Steams Best Practice

Question

Hi,

My customer currently has a large DynamoDB table with DynamoDB Streams enabled and two Lambda consumers of the Stream.  They wish to be able to handle up to five additional consumers, but are aware of [potential throttling if more than two consumers are reading from the same stream][1].  They are considering using a Kinesis Data Steam for DynamoDB to fan-out the events to the additional consumers, they have raised the following questions:

1. Is it good practice to add the Kinesis Data Steam for DynamoDB with the existing DynamoDB Stream and two Lambda consumers still in place, or should they look at removing the existing DynamoDB Stream and use the Kinesis Data Steam for DynamoDB for all consumers?
 2. Any pitfalls they should be aware of when using a Kinesis Data Steam for DynamoDB vs DynamoDB Streams?  Ones I can thin of are:
1. When using Kinesis Data Steam for DynamoDB records may appear in a different sequence than the actual modifications to the item, where in a DynamoDB Steam the records appear in a time-ordered sequence
2. The number of Kinesis Data Steam Shards will need to be managed when using Kinesis Data Steam for DynamoDB where this is managed for you when using DynamoDB Streams.

Any guidance much appreciated.

Thanks

[1]: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-dynamodb-streams

Accepted Answer

To help answer your questions:

1. Its always better to create 1 stream from DDB, either DDB streams or a Kinesis. So if your customer is planning to use more than 2 consumers, I would suggest them to move to Kinesis streams and then have all their Lambda consume from Kinesis. You also get the advantage of data retention over DDB streams.
 2. Besides the ones that you have mentioned:
     - a. Stream records may appear more than once in the Kinesis data stream. You can use the timestamp attribute ApproximateCreationDateTime on each record to identify the actual order in which item modifications occurred, and to identify duplicate records in the stream.
     - b. Using Kinesis Data Streams for DynamoDB is subject to both Kinesis Data Streams pricing for the data stream and DynamoDB pricing for the source table. So beware of the additional costs involved of Kinesis plus DynamoDB charges for change data capture for Kinesis Data Streams. They have to also calculate the red. # of shards reqd based on their DDB table.

Dynamo DB Kinesis Steams Best Practice

관련 콘텐츠