By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Dead letter queues, SQS events and network issues

1

Regarding DLQs, according to https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html: "The maxReceiveCount is the number of times a consumer tries receiving a message from a queue without deleting it before being moved to the dead-letter queue. Setting the maxReceiveCount to a low value such as 1 would result in any failure to receive a message to cause the message to be moved to the dead-letter queue. Such failures include network errors and client dependency errors."

I'm confused by the wording "consumer tries receiving". In the situation where a lambda is receiving messages from a source SQS queue via an event trigger, what is the "consumer" ... the lambda, or the AWS internal service which polls the source queue and sends any found messages to the lambda?

What types of network errors would cause AWS to move a message from the source queue to the DLQ?

Thanks in advance.

Topics
ServerlessComputeApplication Integration
Tags
AWS LambdaAmazon Simple Queue Service
Language
English
CharlesNelsonReilly
asked 4 months ago165 views
1 Answer
0
Accepted Answer

In the context of Amazon SQS and AWS Lambda, the "consumer" refers to the AWS Lambda function that is triggered by the SQS event and processes the messages from the queue.

When a message is received from the SQS queue, the Lambda function attempts to process it. If the processing is successful and the message is deleted from the queue, the process ends. However, if the Lambda function fails to process the message (due to an error in the function, for example), the message remains in the queue and the "ReceiveCount" for that message is incremented, until maxReceiveCount is reached, then going to DLQ.

Network errors that could cause a message to be moved from the source queue to the DLQ include issues with the network connectivity between the AWS services (SQS and Lambda), or within the Lambda function's execution environment. It's important to note that the SQS service itself is designed to be highly available and resilient to network errors, so network errors within SQS itself are rare.

If your lambda process multiple messages at a time, you should implement 'partial batch responses' so that not all messages are put back in the queue if one fails. Look for 'Implementing partial batch responses' in AWS Lambda SQS Doc, you will find some code samples.

profile pictureAWS
Julien Coupez
answered 4 months ago
profile picture
EXPERT
MaxCloud
reviewed 4 months ago
  • CharlesNelsonReilly
    4 months ago

    Thank you for your reply. When you say "If the processing is successful and the message is deleted from the queue", is the message deleted manually by the lambda, or automatically by AWS when the lambda returns successfully (ie, does not throw an exception)?

  • Julien Coupez
    4 months ago

    When you implement partial batch responses, every message that are NOT in the "batchItemFailures" array are considered as proceed and removed from the queue. If your function throws an exception, the entire batch is considered a complete failure and messages are requeued after the visibility timeout.

  • CharlesNelsonReilly
    4 months ago

    One other question. Re "if the Lambda function fails to process the message (due to an error in the function, for example), the message remains in the queue and the "ReceiveCount" for that message is incremented, until maxReceiveCount is reached, then going to DLQ."

    What causes the repeated incrementing of the ReceiveCount? Repeated attempts to get the lambda to process the message?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content