SQS - FIFO Queue - Exactly Once Processing

0

For FIFO Queue, If the visibility time-out expires while a consumer is still processing a message, will it become available and can be returned again and therefore could it be processed twice ? If not, how is it prevented to happen to ensure exactly once processing ?

DP
質問済み 4ヶ月前259ビュー
3回答
1

Hello.

You may want to read the blog below.
If the visibility time-out is exceeded, messages from other consumers can be processed, so in order to avoid processing those messages, add a process to the consumer that writes the message to another storage and checks whether it is duplicated.
https://sookocheff.com/post/messaging/dissecting-sqs-fifo-queues/

Concurrent Consumers: Visibility Timeout Error

The problem is even worse than that: even with a single consumer, if you process a message and then fail to delete the message from SQS within the visibility timeout, the message will be delivered twice. One solution to guard against this would be for the consuming system to read messages from SQS into their own durable storage for future processing. After confirming that a copy of the message has been durably committed, the consumer can delete the message from SQS. This concept is similar to how many database systems use a write-ahead-log to durably store commits. This scenario is depicted in the figure that follows: after each message is committed to the database, it is deleted from the SQS queue, while in parallel a message processor running as a background thread handles processing the message.

profile picture
エキスパート
回答済み 4ヶ月前
1

It uses a message deduplication ID https://docs.aws.amazon.com/sns/latest/dg/fifo-message-dedup.html

If a message with a particular deduplication ID is successfully published to an Amazon SNS FIFO topic, any message published with the same deduplication ID, within the five-minute deduplication interval, is accepted but not delivered. The Amazon SNS FIFO topic continues to track the message deduplication ID, even after the message is delivered to subscribed endpoints.

profile picture
エキスパート
Steve_M
回答済み 4ヶ月前
1

SQS FIFO guarantees that you will get each message once, only if you process it and delete it during the visibility timeout. If you do not delete the message or extend the visibility timeout, the message will be available for other consumers.

This is different than standard queue, where two consumers may get the same message, even if the visibility timeout did not expire yet.

profile pictureAWS
エキスパート
Uri
回答済み 4ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ