By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Determine source of message in DLQ

0

We are looking at ways of unifying our DLQ handling but a problem we are facing an issue with identification of the actual source. Our current approach is to have a unique DLQ for each lambda or similar (could be sns, scheduler, etc.) and then have a lambda with the DLQ as source which further processes the failed messages (could be automatic retries, storing in Dynamo for manual handling, etc.). But we think this becomes a bit cumbersome since we will quickly incur quite high fees for the polling (we have a lot of places where we need DLQs). Our absolute dream scenario is if we could set a tag where we add the DLQ but this isn't available. An alternative would be to use the SenderID in the SQS message to identify the source but this doesn't really seem possible.

Anyone have any suggestions?

Topics
ServerlessComputeApplication Integration
Tags
AWS LambdaAmazon EventBridgeAmazon Simple Queue Service
Language
English
Lewenhaupt-Invoier
asked a year ago290 views
1 Answer
0

One potential solution for identifying the source of a message in a DLQ could be to include metadata in the original message that identifies the source. This could be done by adding a custom header or a JSON field to the message that includes information about the source. This metadata could then be used to determine the source of the message when it ends up in the DLQ. Another potential solution is to use the SNS topic ARN as a way to identify the source, since it will be included in the SQS message when it's delivered to the DLQ. Additionally, you could also use CloudWatch Logs to trace the source of the message.

profile picture
Victor van Liederen
answered a year ago
  • Lewenhaupt-Invoier
    a year ago

    Yeah we can determine the source of an Eventbridge event via the actual source. But that doesn't tell us which lambda was the reason it got put in the DLQ. We could of course add some info to the exception that helps us determine that further, but that only works for exceptions origination from our code, not oom issues or similar. Eventbridge scheduler for example adds a lot of useful metadata when it puts items on its DLQ but the same doesn't go for Lambdas. The only useful additional metadata is the RequestId, but that is only useful for finding the execution if you already know the lambda.

    Yeah we thought about the SNS Topic ARN, but seeing as we can't push from SQS to SNS without having a lambda in between that doesn't solve the solution. It does work when we use Lambda destinations though, but not in other scenarios.

  • Lewenhaupt-Invoier
    a year ago

    I've just realised that we get a lot more info if we use the DLQ as a Lambda Destination instead of directly as a DLQ.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content