Need Help Ensuring Message Order in AWS SQS FIFO Queue

0

Hello AWS Community,

I am currently working on a project where maintaining the order of messages in an SQS queue is critical. I'm using an SQS FIFO queue with the expectation that messages will be processed in the exact order they are sent. However, I'm encountering an issue where the order of the messages in the queue is not being preserved as expected.

For example, if I send five messages in the sequence [1, 2, 3, 4, 5], they do not always appear in the queue in this order. Instead, I see them in orders such as [2, 3, 4, 5, 1] or even with duplicates like [1, 1, 2, 3, 4].

Here is a snippet of my code where I send messages to the SQS FIFO queue:

queueProducerUtility

import { Producer } from "sqs-producer";
import { SQSClient } from "@aws-sdk/client-sqs";
import { v4 as uuidv4 } from "uuid";

const queueUrl = process.env.QUEUE_URL;
const config = {
  region: process.env.REGION,
  credentials: {
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.SECRET_ACCESS_KEY,
  },
};

// create simple producer
const producer = Producer.create({
  queueUrl: queueUrl,
  region: "us-east-1",
  sqs: new SQSClient(config),
});

const sendMessageToSQS = async (message) => {
  const response = await producer.send({
    id: uuidv4(),
    body: JSON.stringify(message.body.content),
    groupId: message.headers.customerChannelIdentifier,
    deduplicationId: uuidv4(), // typically a hash of the message body
  });
  return response;
};

export default sendMessageToSQS;

messageController

import RESPONSE from "../common/http-status-codes.mjs";
import AWSService from "../services/awsService.mjs";
import producer from "../utilities/queueProducerUtility.mjs";
import { Logger } from "../logger/logger.mjs";

const logger = Logger("Message-Controller");

const pushMessageToQueue = async (req, res) => {
  try {
          const _awsService = new AWSService(req);
          console.log("Mesaage Content : Producer  ", req.body.body.content);
          // const response = await producer(req.body);
          const sqsResponse = await _awsService.sendMessageToSQS();
        RESPONSE.OK(sqsResponse, res);
  } catch (error) {
          logger.error("Error in pushing message to Queue:", error);
  }

};

export default {
  pushMessageToQueue,
};
1回答
1
承認された回答

How do you call pushMessageToQueue? Do you wait for it to complete before calling it again?

SQS FIFO preserves the order of messages as they are put into the queue, so you must make sure that you actually put them in order. For that you need to do it one after the other.

With regards to duplicates, you can use message deduplication to ensure that even if the producer tried adding the same message twice, it will appear in the queue only once.

profile pictureAWS
エキスパート
回答済み 6ヶ月前
profile picture
エキスパート
レビュー済み 6ヶ月前
  • Yes i am using await so that it complete its execution before sending message again. you can check my code snippets

  • the issue is i am using group_Id that is used for scalling. After removing that all is working fine. thanks

  • SQS FIFO maintains order within all messages with the same message group ID, so you must include one and use the same for all messages that should be in order.

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ