Why are the asynchronous event sources taking a long time to trigger my Lambda function?

Short description

When using asynchronous invocations with your Lambda functions, you might notice that your events take longer than expected to trigger. Or, your events might be sent to the dead-letter queue (DLQ) without being picked up by the Lambda service for further processing.

There are several different AWS Services that invoke Lambda functions asynchronously:

• Amazon Simple Storage Service (Amazon S3)
• Amazon Simple Notification Service (Amazon SNS)
• Amazon Simple Email Service (Amazon SES)
• AWS CloudFormation
• Amazon CloudWatch Logs
• Amazon CloudWatch Events
• AWS CodeCommit
• AWS Config

Resolution

Invocation delays

Generally, you might see delays due to the asynchronous method of handling requests.

• When you make a request, it's sent to an internal asynchronous queue. Lambda picks up requests from the internal asynchronous queue, and then sends the request to the Lambda function for further processing. You might see a delay in the Amazon CloudWatch logs for your function if your Lambda function runs into errors. Delays can also occur if the requests are throttled at function level.
• If there are no errors or throttle data points at the function level, then check the Regional level metrics for errors and throttles. If other functions are invoked asynchronously and then encounter errors, then these functions are significantly delayed. So, even if your function doesn't have error or throttle data points, the invocations or requests made to the function might still be delayed.
• If other functions that are in the same Region as your Lambda function invoke a number of times asynchronously, then the internal queue becomes congested. To resolve the issue, increase the Regional level concurrency.
• To check the internal queue backlog, turn on AWS X-Ray on your Lambda function. When AWS X-Ray is turned on, you can use the dwell time property. This property shows the total time that requests spend in the internal queue before the Lambda service sends them to the Lambda function for processing.
• Set up a per-function concurrency, or reserved concurrency, to protect functions against the queue backlog.

Duplicate invocation

Lambda is a distributed service, and it makes sure that your function is invoked at least once. However, if the function is invoked more than once, then duplicates can occur. Check the CloudWatch logs for these duplicate invocations.

• It's a best practice to make sure that your function can handle duplicate requests. For more information, see How do I make my Lambda function idempotent?
• Review the CloudWatch logs for Lambda, and check the request IDs for your function. From the logs, you can check whether duplicate events have the same or different request IDs. Request IDs remain the same throughout the lifecycle of an asynchronous invoke for Lambda. If the Request IDs are the same, then check whether the function has any error data points that caused the invocation to retry and duplicate.
• If the Request IDs are different, then the duplicate invocations are occurring on the client side.

Missing invocations

Review the CloudWatch logs to differentiate between missing and delayed invocations. For delayed invocations, follow the steps previously detailed in the Delayed invocations section of this article.

Missing invocations occur when there isn't enough concurrency to serve the request. If the function has reserved concurrency and the function runs into errors, the request spends a long time in the asynchronous queue. The request is then deleted without being processed by Lambda. Review the AsyncEventsDropped metric to check the number of events that were dropped without running the function.

If you configured DLQ, check the DLQ or the on-failure destination for the request. If an event expires in the internal queue after six hours, then the request can be sent to the DLQ without being processed by Lambda.

Metrics for asynchronous invocation

For more information on reviewing further metrics for asynchronous invocation, see Introducing new asynchronous invocation metrics for AWS Lambda.

Related information

Asynchronous invocation

Handling Lambda functions idempotency with AWS Lambda Powertools

Working with Lambda function metrics