Why is my Lambda function retrying valid Amazon SQS messages and placing them in my dead-letter queue?

Short description

If your Lambda function is throttled, returns an error, or doesn't respond when reading an Amazon SQS message batch, then the messages return to your queue. After the visibility timeout occurs, your Lambda function receives the message batch again. If your function fails to process valid messages multiple times, then Amazon SQS sends the messages to your dead-letter queue, if you've configured one.

To prevent valid messages from being placed in a dead-letter queue, your function code must be idempotent and capable of handling messages multiple times. For more information, see How can I prevent an Amazon SQS message from invoking my Lambda function more than once?

Resolution

Verify that your Lambda function's code is idempotent

For idempotency best practices and example function logic, see How do I make my Lambda function idempotent?

Verify that your Amazon SQS queue's visibility timeout is at least six times longer than your Lambda function's timeout setting

Set your source queue's visibility timeout to at least six times longer than your function's timeout. The extra time allows your function to retry processing a batch if the function is throttled while processing a previous batch.

For more information, see Setting the visibility timeout in the Amazon SQS Developer Guide.

Note: If your function doesn't receive messages because the queue's visibility timeout isn't long enough, then the messages won't be recorded in your Amazon CloudWatch Logs.

Verify that the maxReceiveCount attribute is set to at least five on your source queue's redrive policy

Set the maxReceiveCount on the source queue's redrive policy to at least five. If your function returns an error, or can't be invoked because it's at maximum concurrency, processing might succeed with additional attempts. A maxReceiveCount of at least five gives your messages more chances to be processed before they're sent to your dead-letter queue.

To update the maximum concurrency setting limit, see Configuring maximum concurrency for Amazon SQS event sources.

For more information, see How do dead-letter queues work? and avoiding inconsistent message processing in the Amazon SQS Developer Guide.

Check the Lambda function for throttling and reserved concurrency

Lambda functions are sometimes throttled to protect your resources and downstream applications. Even though Lambda automatically scales to accommodate incoming traffic, your function can still be throttled for various reasons.

Use the Lambda console to check the setting for reserved concurrency. If reserved concurrency isn't configured, then the function uses unreserved concurrency. When invocations with functions exceed the unreserved concurrency, throttling occurs.

Note: If you configured a function with zero reserved concurrency, then the function is throttled because it can't process any events. Make sure that you increase the value to a number greater than zero.

If other functions in the same AWS Region are using the concurrency limit, throttling can occur. Lambda functions experiencing throttling send the SQS messages to the dead-letter queue.

Avoid reprocessing all SQS messages in a failed batch

All SQS messages are visible in the dead-letter queue for Lambda functions encountering errors when processing a batch, including messages that Lambda processed successfully. Lambda then retries the entire batch of SQS messages again.

To avoid reprocessing all SQS messages in a failed batch, configure your event source mapping with the value ReportBatchItemFailures in the FunctionResponseTypes list. This lets your function return a partial success and reduce the number of retries. For more information, see Reporting batch item failures.

Identify and resolve any errors that your Lambda function returns

Follow the instructions in How do I troubleshoot Lambda function failures? Your function automatically removes messages from your queue only if the function doesn't return an error.

Related information

How do I request a concurrency limit increase for my Lambda function?

How do I troubleshoot Lambda function throttling with "Rate exceeded" and 429 "TooManyRequestsException" errors?