You can think of the KCL as a micro-batch system. An instance of the KCL will take a micro-batch of records from a single shard in a Kinesis Stream and then one of two things happens: either it is successful in processing those records or it is not. If it is successful, it will move on and get the next micro-batch of records from the stream. If it is not successful, the system will attempt to re-process the same micro-batch.
If the KCL is interrupted for some reason (eg an EC2 scale event, EC2 failure etc), then it will not be successful in processing the current micro-batch. The micro-batch will therefore be re-processed by another EC2 instance that is running the KCL.
When you start the KCL, it creates a table in DynamoDB that tracks the state of the processing against each shard in the Kinesis Stream. This includes a lease system, so the overall system can know if an agent that is processing records from a shard has stopped responding and the batch needs to be reassigned to another agent.
GStreamer plugin for Kinesis data streamasked 4 days ago
Can the customer check how their data are currently partitioned in Timestream?asked 3 months ago
Kinesis data stream - data transfer charges for on-premises consumer/producerAccepted Answerasked 4 years ago
How do we make data un-usable for a AWS Data Exchange customer when the subscription ends?asked 4 months ago
Multiple Kinesis Data Analytics apps to use the same Kinesis firehose delivery stream as sourceAccepted Answerasked 2 years ago
Question on how KCL creates new RecordProcessors with multiple instancesasked 2 years ago
Kinesis data stream iterator age spikesasked 2 months ago
I have question on KCL(Kinesis Customer Library)Accepted Answerasked 4 years ago
How do i check incoming OPC UA dataAccepted Answerasked 9 months ago
Kinesis Transformation Buffering from Data StreamAccepted Answerasked 4 months ago