From AWS docs:
"The most common cause of skipped records is an unhandled exception thrown from processRecords. The Kinesis Client Library (KCL) relies on your processRecords code to handle any exceptions that arise from processing the data records. Any exception thrown from processRecords is absorbed by the KCL. To avoid infinite retries on a recurring failure, the KCL does not resend the batch of records processed at the time of the exception. The KCL then calls processRecords for the next batch of data records without restarting the record processor. This effectively results in consumer applications observing skipped records. To prevent skipped records, handle all exceptions within processRecords appropriately."
Our KCL client is written in node, following this example.
No matter how we tried crashing it, the KCL function simply exits and next time we start it starts from the SAME checkpoint. In other words the above statement doesn't seem to hold true. It does not skip records.
For our application we can't afford dropped records and need to be 100% sure this can't happen.
Can someone with more kinesis/aws experience comment how exactly the above can happen on the consumer side (we're handling the producer already)?
AWS OFFICIALUpdated 2 years ago
