By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Implementing DLQ for both network issues and lambda processing failure

0

Hello. I have a lambda function which has an SQS queue event source. I'd like to have messages sent to the DLQ when one of two things occurs: (1) When the lambda has determined that the message cannot be processed (eg for violating some business rule, etc.), and (2) when the message event cannot be sent to the lambda due to some resource issue such as the network being down.

In the first case, I'd like the message to be sent to the DLQ immediately. In the second case, I'd like the message to be retried until either the lambda becomes available, or some maximum number of retries has occurred.

I understand how the second case can be addressed, via redrive policy with the source queue's deadLetterTargetArn pointing to the DLQ. And I know about the redrive maxReceiveCount value.

It's unclear to me whether the redrive policy also covers the first case, where the lambda determines via code logic that the message should immediately be sent to the DLQ. If this is not covered by the redrive policy, does the lambda just forward the message to the DLQ itself?

Thanks in advance.

Topics
ServerlessComputeApplication Integration
Tags
AWS LambdaAmazon Simple Queue Service
Language
English
CharlesNelsonReilly
asked 4 months ago213 views
2 Answers
0
Accepted Answer

It's unclear to me whether the redrive policy also covers the first case, where the lambda determines via code logic that the message should immediately be sent to the DLQ. If this is not covered by the redrive policy, does the lambda just forward the message to the DLQ itself?

As you need to verify a "violating some business rule", it is preferable to have a Lambda logic in the code and then send the message to the DLQ. Reading through the document below, I did not find any mechanism to have redrive policy that can validate a custom business logic.

[+] Amazon SQS dead-letter queues - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html [+] Understanding SQS retries - https://docs.aws.amazon.com/lambda/latest/operatorguide/sqs-retries.html

AWS
SUPPORT ENGINEER
Sanath_M
answered 4 months ago
profile picture
EXPERT
Gary Mclean
reviewed a month ago
0

For the first case, as this is something you identify in your own code, why not just send a message to the DLQ from within the Lambda function?

profile pictureAWS
EXPERT
Uri
answered 4 months ago
profile picture
EXPERT
Gary Mclean
reviewed a month ago
  • CharlesNelsonReilly
    4 months ago

    Thanks for your reply.

    This is the approach I'm taking, but I'm not sure if this is the AWS recommended approach. Is there a native AWS process that one can use for retries triggered by the lambda, similar to the process where one associates a DLQ with a business queue via the queue's redrivePolicy and deadLetterTargetArn attributes?

  • Uri EXPERT
    4 months ago

    There is a feature in Lambda which is called Lambda Destinations. It behaves differently for different event sources. For instance, for asynchronous invocations, you can specify a failure destination or a success destination. For event source mapping, but not all of them, there are different failure destinations. SQS does not have this mechanism. In case the Lambda invocation fails, the message is returned to the queue and handled according to the queue's configuration.

    This is why your best option is to send the message directly from within your code to the DLQ in case of a business failure.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content