occasionally losing event/stream from kinesis to lambda

Question

I am relatively new to AWS. I am working on a system that uses a single Kinesis stream to trigger 2 lambdas. In most cases this is working but occasionally I encounter cases (a few daily) where one or the other lambda fails to process one of the events even though it is clear it has been processed in the other lambda. What might be the cause of this and how would I go about troubleshooting/fixing it?

Answer

Is the second lambda that was not able to process the message from KDS getting the message or not getting at all? Are you able to tell from the Lambda logs?

If the lambda got the message but encountered a failure, then what are your settings for Retry Attempts and have you configured "Split Batch on Error" and "On Failure Destination". Please refer to this - https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html

If the lambda doesn't get the message at all, is it possible that the 2MB/second/shard limit has been exceeded? Have you configured the lambdas as Enhanced Fanout consumers.

Another thing to check will be do the lambdas have enough concurrency available in the account to handle the number of shards on the Kinesis Streams and in case you have configured "Concurrent Batches Per Shard" lambda will need even higher concurrency.

occasionally losing event/stream from kinesis to lambda

Relevant content