SQS - FIFO Queue - Exactly Once Processing

0

For FIFO Queue, If the visibility time-out expires while a consumer is still processing a message, will it become available and can be returned again and therefore could it be processed twice ? If not, how is it prevented to happen to ensure exactly once processing ?

DP
demandé il y a 4 mois259 vues
3 réponses
1

Hello.

You may want to read the blog below.
If the visibility time-out is exceeded, messages from other consumers can be processed, so in order to avoid processing those messages, add a process to the consumer that writes the message to another storage and checks whether it is duplicated.
https://sookocheff.com/post/messaging/dissecting-sqs-fifo-queues/

Concurrent Consumers: Visibility Timeout Error

The problem is even worse than that: even with a single consumer, if you process a message and then fail to delete the message from SQS within the visibility timeout, the message will be delivered twice. One solution to guard against this would be for the consuming system to read messages from SQS into their own durable storage for future processing. After confirming that a copy of the message has been durably committed, the consumer can delete the message from SQS. This concept is similar to how many database systems use a write-ahead-log to durably store commits. This scenario is depicted in the figure that follows: after each message is committed to the database, it is deleted from the SQS queue, while in parallel a message processor running as a background thread handles processing the message.

profile picture
EXPERT
répondu il y a 4 mois
1

It uses a message deduplication ID https://docs.aws.amazon.com/sns/latest/dg/fifo-message-dedup.html

If a message with a particular deduplication ID is successfully published to an Amazon SNS FIFO topic, any message published with the same deduplication ID, within the five-minute deduplication interval, is accepted but not delivered. The Amazon SNS FIFO topic continues to track the message deduplication ID, even after the message is delivered to subscribed endpoints.

profile picture
EXPERT
Steve_M
répondu il y a 4 mois
1

SQS FIFO guarantees that you will get each message once, only if you process it and delete it during the visibility timeout. If you do not delete the message or extend the visibility timeout, the message will be available for other consumers.

This is different than standard queue, where two consumers may get the same message, even if the visibility timeout did not expire yet.

profile pictureAWS
EXPERT
Uri
répondu il y a 4 mois

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions