SQS Message retention period - how can we send old messages to dead letter queue?

Question

We thought that **Message retention period** is supposed to impact sending this message to Dead Letter Queue..
We were surprised to find out that only way to get into dead letter queue is by **Maximum receives**.

When we have "traffic jam" in the queue, our messages don't necessarily reach the Maximum receives. 
Couldn't your message passed to dead letter queue be similar to car's insurance rules? - either 3 years (Message retention period) or 80K Kilometers (Maximum receives) **whatever comes first**

* How do u suggest to handle those old messages when we get a "traffic jam"? 
* just extending the Message retention period..?
* Do you plan to provide a way to mark that we want old messages to be moved to dead letter queue instead of deleting them?

Pandu · Answer

Hi Michal Tsadok,

Please go through the below documentation and steps it will helps to resolve your issue.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/working-with-messages.html

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html

**Message Retention Period and Dead Letter Queue:**

* **Message Retention Period:** This specifies the duration for which a message will remain in the queue before it is automatically deleted. This does not influence whether a message is sent to a DLQ.
* **Dead Letter Queue (DLQ):** Messages are moved to a DLQ only when the maximum receives threshold is reached.

**Handling Traffic Jams and Old Messages:**
**To handle old messages during traffic jams, consider the following strategies:**

* **Extending the Message Retention Period:** This allows messages to stay in the queue longer, giving more time for processing.
* **Processing Efficiency:** Optimize your message processing logic to handle high traffic more effectively.
* **Monitoring and Alerts:** Set up monitoring to detect when messages are aging and alert you to take action.

There is currently no built-in feature to move messages to a DLQ based on their age. However, you can implement a custom solution using Lambda functions to periodically check the age of messages and move older messages to a DLQ.

Uri Segev · Answer

Message retention should be used when a message is no longer relevant after some time elapsed. You should set the max retention based on that. If message never "expire", you should set the retention to the maximum.

For this reason, it makes sense that expired messages just disappear, as they have no relevance anymore and moving them to the DLQ means that you will still try to process them.

If your traffic jam is so severe that you are not getting to your messages in 14 days (max retention), you should increase the number of consumers to process the messages faster.

Didier Durand · Answer

Hi,

Re DLQ retention policy, have a look at https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html#understanding-message-retention-periods

You may also want to read https://www.ranthebuilder.cloud/post/amazon-sqs-dead-letter-queues-and-failures-handling-best-practices for the best practices re. DLQ management

Best,

Didier

SQS Message retention period - how can we send old messages to dead letter queue?

Add your answer

Relevant content