Need Help Ensuring Message Order in AWS SQS FIFO Queue

0

Hello AWS Community,

I am currently working on a project where maintaining the order of messages in an SQS queue is critical. I'm using an SQS FIFO queue with the expectation that messages will be processed in the exact order they are sent. However, I'm encountering an issue where the order of the messages in the queue is not being preserved as expected.

For example, if I send five messages in the sequence [1, 2, 3, 4, 5], they do not always appear in the queue in this order. Instead, I see them in orders such as [2, 3, 4, 5, 1] or even with duplicates like [1, 1, 2, 3, 4].

Here is a snippet of my code where I send messages to the SQS FIFO queue:

queueProducerUtility

import { Producer } from "sqs-producer";
import { SQSClient } from "@aws-sdk/client-sqs";
import { v4 as uuidv4 } from "uuid";

const queueUrl = process.env.QUEUE_URL;
const config = {
  region: process.env.REGION,
  credentials: {
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.SECRET_ACCESS_KEY,
  },
};

// create simple producer
const producer = Producer.create({
  queueUrl: queueUrl,
  region: "us-east-1",
  sqs: new SQSClient(config),
});

const sendMessageToSQS = async (message) => {
  const response = await producer.send({
    id: uuidv4(),
    body: JSON.stringify(message.body.content),
    groupId: message.headers.customerChannelIdentifier,
    deduplicationId: uuidv4(), // typically a hash of the message body
  });
  return response;
};

export default sendMessageToSQS;

messageController

import RESPONSE from "../common/http-status-codes.mjs";
import AWSService from "../services/awsService.mjs";
import producer from "../utilities/queueProducerUtility.mjs";
import { Logger } from "../logger/logger.mjs";

const logger = Logger("Message-Controller");

const pushMessageToQueue = async (req, res) => {
  try {
          const _awsService = new AWSService(req);
          console.log("Mesaage Content : Producer  ", req.body.body.content);
          // const response = await producer(req.body);
          const sqsResponse = await _awsService.sendMessageToSQS();
        RESPONSE.OK(sqsResponse, res);
  } catch (error) {
          logger.error("Error in pushing message to Queue:", error);
  }

};

export default {
  pushMessageToQueue,
};
1 Answer
1
Accepted Answer

How do you call pushMessageToQueue? Do you wait for it to complete before calling it again?

SQS FIFO preserves the order of messages as they are put into the queue, so you must make sure that you actually put them in order. For that you need to do it one after the other.

With regards to duplicates, you can use message deduplication to ensure that even if the producer tried adding the same message twice, it will appear in the queue only once.

profile pictureAWS
EXPERT
Uri
answered 5 months ago
profile picture
EXPERT
reviewed 5 months ago
  • Yes i am using await so that it complete its execution before sending message again. you can check my code snippets

  • the issue is i am using group_Id that is used for scalling. After removing that all is working fine. thanks

  • SQS FIFO maintains order within all messages with the same message group ID, so you must include one and use the same for all messages that should be in order.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions