SQS messages aren't retained for all retention time when load is high

When dealing with high load in SQS queues, there are several factors that could explain why messages aren't being retained for the full 7-day retention period and aren't appearing in your DLQ.

First, it's important to understand that the message retention period (7 days in your case) is the maximum time a message can remain in the queue if it's not processed. However, this doesn't guarantee that messages will stay in the queue for the full 7 days.

When you have a high volume of messages, the behavior you're experiencing could be due to:

1. Visibility timeout issues: When your Lambda function receives a message, that message becomes invisible for the duration of your visibility timeout (15 minutes). If your Lambda function is taking longer to process messages during high load, or if it's failing without properly handling the message, the message will become visible again after the visibility timeout expires. This counts as one "receive" against your maxReceiveCount (96).

2. Distributed system sampling: SQS is a distributed system that samples its servers when reporting metrics. During high load, this sampling behavior might not give you a complete picture of all messages in the queue, making it appear that messages are disappearing.

3. Batch processing behavior: With a batch size of 10 and "Report batch item failures" enabled, make sure your Lambda function is correctly reporting failed items. If your code isn't properly handling the batch processing, it might be deleting messages that weren't successfully processed.

4. Lambda concurrency and throttling: During high load, if your Lambda function hits concurrency limits, it might not be able to process all incoming messages, potentially causing some to be processed differently than expected.

To resolve this issue:

1. Ensure your Lambda function's timeout is less than the SQS visibility timeout (15 minutes).

2. Implement comprehensive error handling in your Lambda function to properly report batch item failures.

3. Consider increasing the visibility timeout if your processing takes longer during high load.

4. Check your Lambda function's concurrency settings and consider increasing them if necessary.

5. Implement detailed logging in your Lambda function to track message processing, especially during high load scenarios.

6. Verify that your redrive policy is correctly configured to move messages to the DLQ after the specified number of receives.

With these adjustments, your messages should be retained in the queue for the full retention period until they can be successfully processed or moved to the DLQ after the maximum number of receives.
Sources
Why Are SQS Messages Moving to DLQ Without Being Delivered to Lambda? | AWS re:Post
SQS, "Queues" dashboard, NO CONSUMERS, "Messages available" number decreases by thousands, why? | AWS re:Post
SQS messages getting skipped | AWS re:Post

When your queue is packed and processing is slow, messages are likely hitting the visibility timeout (15 minutes) before your Lambda can complete processing them. This causes SQS to make the message visible again for retry, but if the queue remains congested, the same cycle repeats until the message reaches its maximum receives limit (96) and gets sent to the DLQ - all without the full 7-day retention period being utilized. Maximum Lambda timeout: 15 minutes Your SQS visibility timeout: 15 minutes

Possible Solutions:

1. Reduce Batch Size

{
  "BatchSize": 1,  // Process one message at a time
  "MaximumBatchingWindowInSeconds": 0
}

0r 2. Increase Visibility Timeout

• Set visibility timeout to at least 2-3x your maximum expected processing time
• Consider setting it to 30-60 minutes if your processing can take that long

Manoj, this was true if we would have gotten all those messages in DLQ, but we don't get any.. It's rarely get to 96 retries and then we see it in DLQ (rarely => 10 messages out of 20K)

Can you validate your assumption relate to the "silent" retries? we also notice several time that the message is exhausted after 22h (which means that it would even not finish the 96*15min retries)

Is there a way to see the path a message has gone through since it arrived through SNS, during the retry until the time it's deleted / moved to DLQ?

We ensured there is no timed out logs filter @message like /Task timed out/ and Lambda Timeout 15min is aligned to SQS Default visibility timeout 15 Minutes 5 minutes is the max duration in the timeframe where we see that messages aren't retained for 7 days

Waiting for your response Michal

Hello,

Warm Greetings!

From the description, I understand that with the high volume of messages you observed that the messages were not getting retained in the Queue for 7 days (Queue retention period) and messages were also not getting retried as expected.

To dive deep into the issue we need to verify the SQS metrics. In the scenario you can check the 'ApproximateAgeOfOldestMessage' metric this helps us to verify the the age of the oldest unprocessed message in the queue.

ApproximateAgeOfOldestMessage - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html

Additionally you can log the SQS messageIDs in lambda function code to verify the messages received and processed by the lambda function. I would also recommend you to review, if you are having the error handling configured in your lambda function code. As you might already be aware that the message received from SQS queue will return to the queue to be retried only if the Lambda function execution fails with unhandled exception or visibility timeout has expired.

Further if the issue still persists, I would recommend you to reach out to AWS Premium Support by raising a Support Case using the same account with all the information [+]. Kindly note that to identify root cause of the issue we require more information from your end which cannot be shared over public platform.

[+] https://console.aws.amazon.com/support/home#/case/create

Regarding DLQ and Queue Retention Period

If your retention period is 7 days the messages will be dropped from the queue without any notification if they are not consumed within 7 days. These messages do not go to the DLQ (they are directly deleted). Messages in the DLQ are only the ones which the consumer consumed but was unable to delete until the maxReceiveCount is reached. In this scenario, the message would need to be received 96 times before being moved to the DLQ. Here expired messages were not sent to DLQ.

As you might already be aware that the expiration of a message in DLQ is always based on its original (main queue) enqueue timestamp. When a message is moved to a dead-letter queue, the enqueue timestamp is unchanged. The 'ApproximateAgeOfOldestMessage' metric indicates when the message moved to the dead-letter queue, not when the message was originally sent.

For example: Let us assume that a message spends 1 day in the original queue before it's moved to a dead-letter queue. If the dead-letter queue's retention period is 4 days, the message is deleted from the dead-letter queue after 3 days and the 'ApproximateAgeOfOldestMessage' is 3 days. Thus, it is a best practice to always set the retention period of a dead-letter queue to be longer than the retention period of the original queue. Hence kindly verify the DLQ queue retention period and configure it to value >7 days.

I hope you find above information helpful!

we log MessageId when we start working on a message and when the message is reported in "Report batch item failures".

In all the cases that messages weren't retained for 7 days and didn't reach the "Maximum receives" we see pair of start-failure logs - i.e. it was failed and back to the queue everytime, but suddenly it's dropped

ApproximateAgeOfOldestMessage = 199K - it's in seconds, which means 2.5 days - not 7 days

after additional discussion with my colleagues, we found the RCA (on our side)

we have base HandleMessageByMessage Subscriber function which is without try/catch it means that if one message fail, all the rest won't be invoked in addition, each lambda instance handle internal cache for MessageIds, to make sure we don't run MessageId twice on high load, we got to a case that same lambda instance run for more than 15 minutes, i.e. can re-run the same MessageId - and this is why the messages disappeared for us (this base function didn't log this MesssageId skip..)

Thanks for your time (and efforts) We learned a lot from digging the queues and the lambda handling

Michal