SQS Message Consume Speed Decrease When Few Messages are left

The decrease in message consumption speed you're experiencing as the queue nears depletion is likely due to the way Amazon SQS handles message retrieval, particularly with short polling.

When using short polling, which is the default behavior, SQS samples only a subset of its servers to find messages. This sampling approach works well when the queue is full, allowing for quick retrieval of messages. However, as the number of messages in the queue decreases, the likelihood of sampling servers that contain messages also decreases.

This behavior can lead to a situation where, with fewer messages left in the queue, many of your ReceiveMessage requests may return empty, even though messages are still available. This results in the sharp decrease in consumption speed you're observing.

To mitigate this issue and maintain a more consistent message consumption rate, consider the following approaches:

1. Implement long polling: Long polling allows SQS to wait until a message is available before sending a response, reducing empty receives. You can enable long polling by setting a non-zero wait time (up to 20 seconds) for the ReceiveMessage API action.

2. Increase the MaxNumberOfMessages parameter: When calling ReceiveMessage, set this parameter to receive up to ten messages per transaction. This can help maintain throughput even when fewer messages are available.

3. Use batch actions: Utilize SQS batch actions like SendMessageBatch, DeleteMessageBatch, and ChangeMessageVisibilityBatch to process messages more efficiently.

4. Optimize your consumer application: Ensure your application is efficiently processing messages and not introducing unnecessary delays.

5. Monitor and adjust: Use Amazon CloudWatch metrics to monitor your queue's performance and adjust your polling strategy as needed.

By implementing these strategies, particularly long polling, you should be able to maintain a more consistent message consumption rate throughout the entire process, even as the number of messages in the queue decreases.
Sources
Amazon SQS short and long polling - Amazon Simple Queue Service
the message consumption speed is extremely slow | AWS re:Post
Using the appropriate polling mode in Amazon SQS - Amazon Simple Queue Service

Are you using a standard queue or a FIFO queue? How are you consuming the messages? Is it using Lambda or some other compute? Are you using long polling or short polling? How many consumers do you have? What are your consumers doing? Could it be that they are accessing some downstream service which takes longer to respond?

The slowdown likely occurs due to:

1. Message distribution across SQS partitions becoming uneven with fewer messages
2. Increased empty receive attempts as the queue depletes

Solutions:

1. Implement Long Polling

// Set waitTimeSeconds to reduce empty receives
.waitTimeSeconds(20)

2. Optimize batch size

// Increase messages per batch
.maxNumberOfMessages(10)

3. Reduce number of consumers when queue depth is low (< 300,000 messages)

For more details on SQS best practices, see:

• AWS SQS Best Practices
• Enabling Long Polling

The sharp drop in SQS message consumption speed as your queue nears empty is a well-known effect of how SQS distributes messages across its infrastructure. When the queue is full, short polling (the default) quickly finds messages because it samples many servers that are likely to have messages. As the queue depletes, more polling requests return empty, which sharply reduces throughput—even though messages remain in the queue[1][6].

To maintain consistent throughput as the queue empties, follow these best practices:

• Enable long polling: Set ReceiveMessageWaitTimeSeconds to a value up to 20 seconds. Long polling reduces empty receives by waiting for messages to become available, which is especially effective when few messages are left[1][6].
  See: SQS Short and Long Polling documentation

• Increase batch size: Always set MaxNumberOfMessages to 10 (the maximum) for each ReceiveMessage call. This maximizes efficiency, even when the queue is nearly empty[1][3].

• Use batch actions for deletes: Deleting messages in batches (DeleteMessageBatch) reduces API calls and helps avoid hitting in-flight message limits[1].

• Monitor and adjust: Use Amazon CloudWatch metrics to track queue depth and consumption rates, and scale your consumers accordingly[1][3].

• Optimize consumer logic: Make sure your consumers process messages efficiently and avoid downstream bottlenecks[3][4].

Why does this happen?
As SQS messages become sparse, short polling is statistically less likely to find messages on each request. Long polling mitigates this by waiting for messages to arrive, returning them as soon as they are available, which keeps your consumers busy until the queue is truly empty[1][6].

Summary:
Switch to long polling, maximize batch sizes, and monitor your queue. These steps will help you maintain high throughput from start to finish.

For more details, see the SQS documentation on polling.

This approach directly addresses the root cause and provides actionable steps, improving on other answers by clearly explaining the mechanism and linking to relevant AWS documentation[1][6][3].

[1] https://repost.aws/questions/QUufea_XL3QY2HAV9TuLYfmQ/sqs-message-consume-speed-decrease-when-few-messages-are-left [2] https://repost.aws/questions/QUKtgKP_8BQb2db2TbskvkmQ/multiple-sqs-messages-of-the-same-group-in-one-batch [3] https://repost.aws/questions/QUjWWqn7cvTf-vgxW5Vhr6oQ/the-message-consumption-speed-is-extremely-slow [4] https://www.reddit.com/r/aws/comments/145dzlj/sqs_scaling_message_consumers/ [5] https://repost.aws/questions/QUGZxHNyvhQ5iTHHxXmeLVpw/reading-millions-of-messages-from-sqs [6] https://repost.aws/questions/QU00V6VdF2TYCDoQQ2Yeo_ZA/waittime-for-sqs-polling [7] https://www.reddit.com/r/aws/comments/sj2kip/conditionally_ratelimiting_an_sqs_queue_based_on/ [8] https://repost.aws/questions/QUv5XMDHimSP2wG52XcYjvbQ/sqs-with-a-high-volume-of-traffic-what-is-the-starting-time-for-calculating-the-visibility-timeout [9] https://softwaremill.com/amazon-sqs-performance-latency/ [10] https://repost.aws/questions/QUXazPds4_QMyJfyQvjFTOQg/sqs-polling-rate