Does AWS SQS allow consumers to filter and retrieve only specific messages?

0

I have multiple consumers that read messages from an AWS SQS. Each consumer is specifically interested in messages that contain a unique value. Every consumer will have at least one message they are interested in, which includes a unique key-value pair such as JobId = HkjshkHkh88HK. No consumer will be interested in messages intended for other consumers, as they are only looking for the key-value pair associated with their unique ID. How can I implement this scenario efficiently using a single AWS SQS and multiple consumers?

Sachin
asked 10 months ago4151 views
4 Answers
0

You can't do that. The messages will be distributed across all consumers without a way to filter the messages.

You could use EventBridge and create multiple rules, with different filters that will distribute to different queues.

profile pictureAWS
EXPERT
Uri
answered 10 months ago
  • Multiple queues are not considered as an option in the proposed architecture. In the current architecture, it is necessary to have a single AWS SQS that is consumed by several consumers, more than 100 in most cases.

0

Hi, if you have a single queue, all readers can read from this queue wihtout acknowledging in a non-destructive manner. Only the reader interested in it will delete the message after he successfully got it.

This scenario implies useless reads by all readers except 1. So, these additional API calls have to be accounted for. Of course, 1queue per reader would be better from API call perspective. It all boils down to how many of such queues you need.

From https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/step-receive-delete-message.html

Amazon SQS doesn't automatically delete a message after retrieving it 
for you, in case you don't successfully receive the message (for example, 
if the consumers fail or you lose connectivity). To delete a message, you 
must send a separate request which acknowledges that you've successfully received 
and processed the message. Note that you must receive a message before you can 
delete it.
profile pictureAWS
EXPERT
answered 10 months ago
  • What is the meaning of a non-destructive manner in this case?

    When a consumer reads a message from AWS SQS without acknowledging it, will the other consumers also be able to read the same message? Wouldn't that message become invisible to all other consumers while one consumer is already reading it?

0

I don't believe SQS is the right service for this use case. You can consider following alternatives :

  • Amazon SNS that allows message filtering at the subscriber but it might or might not fit your use case as it is based on a pub/sub methodology and not a Producer/Consumer
  • Amazon MSK : you can benefit from full power of Kafka to create topics and direct each consumer to consume from its topic. You can deploy Amazon MSK as well in a serverless manner as well as cluster manner.
  • You can drive a serverless consumer using step-function that receives all the messages and triggers the right consumer based on the message categorization (this means that the consumers are better to be Lambda based). In this way you are pushing the consumption logic to the step function and not SQS.
AWS
answered 10 months ago
profile picture
EXPERT
reviewed 10 months ago
0

To me it seems that Kinesis Data Streams are a more applicable solution here. It allows for multiple readers/consumers for each message; each of which can decide whether to process the message or ignore it.

I'd note that the two services (Kinesis and SQS) have different limits in terms of data and message volumes but it might be worth considering.

profile pictureAWS
EXPERT
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions