Why doesn't my Amazon SQS FIFO queue return all messages or messages in other message groups?

Resolution

When you use FIFO queues, you can't request to receive messages from a specific message group ID. You also can't filter the ReceiveMessage API call to return only messages within one message group at a time.

When you specify a MaxNumberOfMessages parameter in the ReceiveMessage API call, SQS returns as many messages with the same message group ID as possible. The ReceiveMessage API call can also return messages from other available message groups.

For FIFO queues, when you receive messages that belong to a particular message group ID, note these guidelines:

• You must delete the messages in the current message group ID before you can receive more messages within the same message group ID.
• Messages that are received but not yet deleted are in the inflight state. Inflight messages block delivery of available messages within the same message group.
• A common reason that messages are in the inflight state is because the consumer client fails unexpectedly. Another reason is when the downstream AWS Lambda consumer function encountered exceptions or reached its maximum concurrency.
• The ChangeMessageVisibility API can extend or shorten the visibility timeout period for inflight messages.
• You can wait until the message visibility timeout period expires to allow message redelivery.

For all FIFO queues and Amazon SQS quota limits, note these guidelines:

• FIFO queues have a maximum of 20,000 inflight messages.
  Note: Amazon SQS doesn't return error messages that exceed quota limits.
• FIFO queues can process up to 300 transactions per second (TPS) without high throughput activated. Requests over 300 TPS get the "ThrottlingException" error even if messages in the queue are available.
• SQS FIFO queue messages are stored in partitions. Each partition supports up to 300 TPS for send, receive, and delete operations. For more information, see Optimizing partition utilization.
• It's possible for FIFO queues with high throughput to share one partition with multiple message groups. If a message group exceeds 300 TPS for its partition, then messages from other groups on that partition aren't delivered because of throttling.
• FIFO queue high throughput transactions per second varies dependent on the AWS Region. For more information, see Quotas related to messages.

To reduce Amazon SQS throughput limit on FIFO queues, note these guidelines:

• Use the Amazon SQS API batch actions SendMessageBatch, DeleteMessageBatch, and ChangeMessageVisibilityBatch to reduce cost or change up to ten messages. For the ReceiveMessage API, set the MaxNumberofMessages parameter to receive up to ten messages per transaction. For more information, see Amazon SQS batch actions.
• For FIFO queues with high throughput, follow the recommendations to optimize partition utilization. Send messages with same message group IDs in batches. Delete messages or change the message visibility timeout values in batches with receipt handles from the same ReceiveMessage API requests.
• Activate high throughput for FIFO queues.
• For FIFO queues with high throughput, use a MessageGroupId with multiple unique values. This allows for an even distribution across FIFO queue partitions. For more information, see Using the Amazon SQS message group ID.

A FIFO queue looks through the first 20,000 messages to determine the available message group. If message groups in the first 20,000 messages are blocked because of inflight messages, then messages from other groups beyond the first 20,000 won't return. For more information, see Avoid having a large backlog of messages with the same group ID.

Example A

A FIFO queue has a total of 20,001 messages. The first 20,000 messages belong to message group 1, and the last message belongs to message group 2. When you try to receive messages from the queue, you receive a message only from group 1. Any consecutive ReceiveMessage API calls result in empty receives. This happens because FIFO looks through only the messages that belong to group 1, and that group is blocked by the current call.

Example B

A FIFO queue has a total of 20,000 messages. The first 19,999 messages belong to message group 1, and the last message belongs to message group 2. When you try to receive messages from the queue, the first ReceiveMessage call gets a message that belongs to group 1. The second ReceiveMessage call gets a message that belongs to group 2. Any additional ReceiveMessage calls result in empty receives because both groups are now blocked by the current calls.

Related information

Amazon SQS FIFO (First-In-First-Out) queues

Using the Amazon SQS message group ID