I have question on KCL(Kinesis Customer Library)

Question

A customer currently use Kinesis (I guess data stream) in their own IoT service.

Customer do believe that KCL recorded how handling data when EC2 is being scaled in or terminated.
(There is a possibility of terminating EC2 when the data has been handled)

**Q1)** They wonder how does this record handle?

Or

**Q2)** They have to set lifecycle hook when EC2 is being scaled in or there is something that KCL managed this record by itself?

Customer would like to have any example or practice.

Accepted Answer

You can think of the KCL as a micro-batch system.  An instance of the KCL will take a micro-batch of records from a single shard in a Kinesis Stream and then one of two things happens: either it is successful in processing those records or it is not. If it is successful, it will move on and get the next micro-batch of records from the stream.  If it is not successful, the system will attempt to re-process the same micro-batch.

If the KCL is interrupted for some reason (eg an EC2 scale event, EC2 failure etc), then it will not be successful in processing the current micro-batch.  The micro-batch will therefore be re-processed by another EC2 instance that is running the KCL.

When you start the KCL, it creates a table in DynamoDB that tracks the state of the processing against each shard in the Kinesis Stream.  This includes a lease system, so the overall system can know if an agent that is processing records from a shard has stopped responding and the batch needs to be reassigned to another agent.

I have question on KCL(Kinesis Customer Library)

Relevanter Inhalt