Can someone explain the point of MessageGroupId?

0

I need to use FIFO.

What is the point of MessageGroupId?

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html

"The message group ID is the tag that specifies that a message belongs to a specific message group. Messages that belong to the same message group are always processed one by one, in a strict order relative to the message group (however, messages that belong to different message groups might be processed out of order). "

SAY WHAT?!

This is why I create different queues and stuff. So there are different queues with different names and different queues.

Is the point of MessageGroupId to create queues within queues? Why would you ever want to do that without confusing other team members?

Is MessageGroupId required if I want to use FIFO?

This article goes on (kill me):

"Any scenario with multiple producers and consumers increases the risk of inadvertently delivering a duplicate message if a worker doesn't process the message within the visibility timeout and the message becomes available to another worker."

Who engineered this? Please come explain this.

asked 2 years ago2930 views
2 Answers
1

If you need all messages in a FIFO queue to be delivered in strict order, just use the same MessageGroupId for all messages. You can name it "default-group". Having that will work just fine, but remember that a FIFO queue like that will not give out the next available message until the previously received message is deleted -> this is the way such queue guarantees processing in order. So you maximum throughput of a pure FIFO queue like that will be limited.

There are however common use cases where you don't need strict ordering of all messages. Imagine a system that keeps track of user data and provides notification about changes to user data. You may want to get notification of changes about particular user in order, but you may want to process notification about different users in parallel.
If you use a single FIFO queue, all notifications will get processed in sequence, no parallelism is possible.
If you use a queue per user, then with a lot of users you need a lot of queues and you need to call ReceiveMessage on each queue to find if there's anything to process.
You can use a single FIFO queue and provide user id as MessageGroupId. You can then poll for messages from that single queue and SQS FIFO queue will give you message that belong to same MessageGroupId in order, but messages with different MessageGroupIds in parallel.

This re:Invent talk explores more messaging patterns across AWS services: https://www.youtube.com/watch?v=4-JmX6MIDDI

Kuba
answered 2 years ago
0

I didn't know that FIFO queues couldn't subscribe to SNS topics. Never mind on this. It doesn't appear FIFO is ready for prime time. I'll continue using RabbitMQ.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions