Avoiding retries with SQS and Lambda

Question

I have a lambda function with an SQS trigger, configured to process a batch of up to 100 records per invoke. I do not want AWS Lambda to invoke my function more than once for the same batch of records. I'm trying to handle this in several ways:

**Inside function handler code**

Here  I'm catching all exceptions. If processing of one record fails, I will continue to process the remaining records. I am not interested in retrying the failing records at a later time.

**Using DLQ for the SQS queue with maxReceiveCount=1**

Potentially my lambda function could time out or be throttled. Also in these cases I do not want any retries. I'm trying to accomplish this using a configuration similar as shown below.

```
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Example of processing messages on an SQS queue with Lambda
Resources:
  MySQSQueueFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs12.x
      Timeout: 300
      Events:
        MySQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt MySqsQueue.Arn
            BatchSize: 100
            MaximumBatchingWindowInSeconds: 1
  MySqsQueue:
    Type: AWS::SQS::Queue
    Properties:
      MessageRetentionPeriod: 14400
      VisibilityTimeout: 1800
      RedrivePolicy:
        deadLetterTargetArn:
          Fn::GetAtt:
            - "MyDLQ"
            - "Arn"
        maxReceiveCount: 1 # No retries
  MyDLQ:
    Type: AWS::SQS::Queue
```

I tried to force throttling by setting reserved concurrency on the lambda function to 0. I noticed that even if the function was throttled, the records did not go immediately to the DLQ. I'm not sure, but I'm suspecting that AWS Lambda retried the failed batches before moving them to the DLQ, even though I have set maxReceiveCount to 1. Is this expected? Is my setup the right way to avoid retries when using SQS and Lambda?

Answer

Your setup is correct for avoiding retries when using SQS and Lambda. The maxReceiveCount property set to 1 on the RedrivePolicy of the SQS queue ensures that the failed records are moved to the Dead Letter Queue (DLQ) immediately after the first failed attempt. This behavior is independent of whether AWS Lambda retries the batch before moving it to the DLQ.

Regarding your observation that records did not go immediately to the DLQ even if the function was throttled, this behavior may occur if the throttling occurred after the records had been dequeued from the SQS queue but before they could be processed by the Lambda function. In this case, Lambda will retry the function invocation for up to six hours with an exponential backoff. However, if the maximum number of retries (which is one, in your case) is reached, the failed records will be moved to the DLQ.

Note that setting the reserved concurrency for a Lambda function to 0 can be used for testing purposes to ensure that the function is throttled. However, in a production environment, this may not be the best way to ensure throttling, as it could result in a complete halt of your processing. Instead, you can consider setting an appropriate concurrency limit that matches the capacity of your downstream resources, such as your database or other AWS services.

Avoiding retries with SQS and Lambda

Relevant content