Why are the asynchronous event sources taking a long time to trigger my Lambda function?
4 minute read
There's a delay when the asynchronous event sources trigger my AWS Lambda function.
When using asynchronous invocations with your Lambda functions, you might notice that your events take longer than expected to trigger. Or, your events might be sent to the dead-letter queue (DLQ) without being picked up by the Lambda service for further processing.
There are several different AWS Services that invoke Lambda functions asynchronously:
Generally, you might see delays due to the asynchronous method of handling requests.
When you make a request, it's sent to an internal asynchronous queue. Lambda picks up requests from the internal asynchronous queue, and then sends the request to the Lambda function for further processing. You might see a delay in the Amazon CloudWatch logs for your function if your Lambda function runs into errors. Delays can also occur if the requests are throttled at function level.
If there are no errors or throttle data points at the function level, then check the Regional level metrics for errors and throttles. If other functions are invoked asynchronously and then encounter errors, then these functions are significantly delayed. So, even if your function doesn't have error or throttle data points, the invocations or requests made to the function might still be delayed.
If other functions that are in the same Region as your Lambda function invoke a number of times asynchronously, then the internal queue becomes congested. To resolve the issue, increase the Regional level concurrency.
To check the internal queue backlog, turn on AWS X-Ray on your Lambda function. When AWS X-Ray is turned on, you can use the dwell time property. This property shows the total time that requests spend in the internal queue before the Lambda service sends them to the Lambda function for processing.
Set up a per-function concurrency, or reserved concurrency, to protect functions against the queue backlog.
Lambda is a distributed service, and it makes sure that your function is invoked at least once. However, if the function is invoked more than once, then duplicates can occur. Check the CloudWatch logs for these duplicate invocations.
Review the CloudWatch logs for Lambda, and check the request IDs for your function. From the logs, you can check whether duplicate events have the same or different request IDs. Request IDs remain the same throughout the lifecycle of an asynchronous invoke for Lambda. If the Request IDs are the same, then check whether the function has any error data points that caused the invocation to retry and duplicate.
If the Request IDs are different, then the duplicate invocations are occurring on the client side.
Review the CloudWatch logs to differentiate between missing and delayed invocations. For delayed invocations, follow the steps previously detailed in the Delayed invocations section of this article.
Missing invocations occur when there isn't enough concurrency to serve the request. If the function has reserved concurrency and the function runs into errors, the request spends a long time in the asynchronous queue. The request is then deleted without being processed by Lambda. Review the AsyncEventsDropped metric to check the number of events that were dropped without running the function.
If you configured DLQ, check the DLQ or the on-failure destination for the request. If an event expires in the internal queue after six hours, then the request can be sent to the DLQ without being processed by Lambda.