SQS - FIFO Queue - Exactly Once Processing

0

For FIFO Queue, If the visibility time-out expires while a consumer is still processing a message, will it become available and can be returned again and therefore could it be processed twice ? If not, how is it prevented to happen to ensure exactly once processing ?

DP
gefragt vor 4 Monaten262 Aufrufe
3 Antworten
1

Hello.

You may want to read the blog below.
If the visibility time-out is exceeded, messages from other consumers can be processed, so in order to avoid processing those messages, add a process to the consumer that writes the message to another storage and checks whether it is duplicated.
https://sookocheff.com/post/messaging/dissecting-sqs-fifo-queues/

Concurrent Consumers: Visibility Timeout Error

The problem is even worse than that: even with a single consumer, if you process a message and then fail to delete the message from SQS within the visibility timeout, the message will be delivered twice. One solution to guard against this would be for the consuming system to read messages from SQS into their own durable storage for future processing. After confirming that a copy of the message has been durably committed, the consumer can delete the message from SQS. This concept is similar to how many database systems use a write-ahead-log to durably store commits. This scenario is depicted in the figure that follows: after each message is committed to the database, it is deleted from the SQS queue, while in parallel a message processor running as a background thread handles processing the message.

profile picture
EXPERTE
beantwortet vor 4 Monaten
1

It uses a message deduplication ID https://docs.aws.amazon.com/sns/latest/dg/fifo-message-dedup.html

If a message with a particular deduplication ID is successfully published to an Amazon SNS FIFO topic, any message published with the same deduplication ID, within the five-minute deduplication interval, is accepted but not delivered. The Amazon SNS FIFO topic continues to track the message deduplication ID, even after the message is delivered to subscribed endpoints.

profile picture
EXPERTE
Steve_M
beantwortet vor 4 Monaten
1

SQS FIFO guarantees that you will get each message once, only if you process it and delete it during the visibility timeout. If you do not delete the message or extend the visibility timeout, the message will be available for other consumers.

This is different than standard queue, where two consumers may get the same message, even if the visibility timeout did not expire yet.

profile pictureAWS
EXPERTE
Uri
beantwortet vor 4 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen