1-3 day latency in getting events

Question

Configuration: DynamoDB streams (delete event - ttl 90 minutes) => lambda => Event Bridge => Kinesis Firehose (Delivery stream with Direct PUT => S3 bucket (firehose destination) => SQS (triggerd on object create) => lambda X

We are seeing 1-3 days delay in the lambda function X from being invoked. In some cases we are not seeing it invoked. No errors. No errors in the kinesis error logs. It looks like object is getting created after a delay in s3 bucket. Max retry is 24 hours. Where could the delay occur? Whats the best way to troubleshoot this?

Accepted Answer

You did not specify what the delay is compared to. Is to to the time the item is removed from DDB?

The first place where delays can occur is in the DynamoDB streams. According to the [documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html), items will be deleted from the table up to 48 hours after the time define in TTL.

The next possible delay is invocation of the Lambda function reading from DynamoDB streams in case there are many items in there. Items will be retained in the stream up to 48 hours. If you do not process them in a timely manner or you have errors in your function, it may prevent the newly added records to be processed. I recommend that you look at the IteratorAge to find out if there are delays processing the records in the stream.

Another place for delay is in Kinesis Firehose which depends on the configuration, can buffer messages up to 15 minutes before writing them to S3 (I know it can't account for the 1-3 days delay, but adding it here for completeness).

And finally, according to [documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html), event notifications from S3 to SQS can sometimes take minutes or longer.

1-3 day latency in getting events

Relevant content