SQS Queue Ring Buffer

0

Hi, I'd like to know if there is a way to support ring buffer functionality to SQS queues?

My infrastructure has multiple usecases of just requiring a latest data snapshot across a variety of apps. The ideal scenario is if I can set a maximum queue size to 1 message that all consumers have read access to, while the queue (or some other AWS piece) can manage the queue size to only contain the latest snapshot.

Currently, the only way I have found to support my use case is to have one queue by consumer, which will become very difficult to manage properly. The ring buffer functionality would effectively solve this issue.

Thanks in advance!

asked 23 days ago64 views
4 Answers
0

Sounds like you're fanning out from one producer to multiple consumers? SNS or Kinesis Data Streams would probably suit.

answered 22 days ago
  • Yes, currently I am using SNS for the fanout to multiple SQS queues via a filter. The purpose for the queue in my use case is to store the latest state.

    My use case is a financial market data related application, which typically uses a messaging bus + ring buffer functionality because applications processing the data are not meant to process every type of event. So far, I found SNS + SQS to similarly mimic this, but it comes with some drawbacks.

    I am not too familiar with Kinesis - does it have this type of functionality?

    (Disclaimer, I'm new to AWS. My former firm managed this in house).

0

I am not sure what exactly you are trying to achieve. A ring buffer is a queue with a limited size. When it is full, no new elements can be added to the queue. There is a different mode in which new elements overwrite existing elements in the queue. In this form, the queue only holds the most recent N items. If I understand correctly, you want to the later implementation and you want the size to be 1, i.e., you want a queue that always holds only the most recent item.

If this is the case, why use a queue? You could just update an item in DynamoDB. Every time a new item comes in, you update the item in the database to the latest value. When you want to consume elements, you just read from the database. If you need to know that you are handling a new item and not the one you already handled before, you can add an additional field which indicates that the item is new or not. The producer sets it to New. The consumer sets it to Old.

profile picture
EXPERT
Uri
answered 22 days ago
  • Market data is known to overload databases, so a DynamoDB cannot be the approach unfortunately. There are companies that make specialized DB's for financial data and even those cannot be used for real time processing. For example, one financial product would post <10kb message every 25ms in normal conditions and consumers would need access to that data as low latency as possible.

    What I am looking for is a multi producer & multi consumer structure where consumers only require the most recent item that does not rely on a database. How are other trading applications on AWS doing this?

0

If I understand well you have multiple consumers who wait for same message buy at a given time they need the last state of event,

Is it the case?

answered 22 days ago
  • For one producer, I will have multiple consumers of the data. The consumers will only care about the latest state if there are multiple messages produced during the time the consumer is processing the change in state.

    As I do more research, I am finding that AWS's MSK solution is most likely the best fit for my needs.

0

Hi Thanks for your replay , now it s clear

In my opinion you dont need a circular queue ad basically you are deduplicating the events at a given time,

**Example : **

*considering your event looks like *

{ Identifier: "A", State: "Created", .... }

and other one

{ Identifier: "A", State: "Approved", .... }

You dont care about the state and you need just At a given time the consumer get Id: "A" one time. In your design you need to use a systeme to deduplicate that events and avoid inserting the new ones if one exists already

You have two choices based on level of complexity of your event structure:

1- Using A FIFO SQS which let you grouping the messages and deduplicating them based on a provided DeduplicationId https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html

2- Use a dynamodb table to deduplicate your event before getting to your SNS topic , i already used this pattern but as Outbox pattern but deduplicated as well is an state in that system, so i avoid generating events , simply using a simple TTL before Put Item and use conditional Expressions to avoid inserting if already exists

behind that the Dynamodb streams trigger a lambda which prepare the event and push it to the SNS topic

Which one to choose? If the deduplication is a concern in your consumer part and not in your own processing i propose using the SQS FIFO , In the contrary if you care also about this problem in your processing as well i propose the Dynamodb Table solution

BTW its upon your need and design

**A final word: In a fanout design avoid taking ownership of consumer Queues but if that part is in your Bounded context as well non problem **

Having a look at following topics will help :

Message Filtering Pattern

Serverless Communication Patterns

answered 21 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions