Skip to content

Using a Common SQS for Triggering Deployments individually

0

Hello,

Is it possible to use a single SQS queue to trigger multiple deployments, but not simultaneously?

For example, we have a ScaledObject configured for multiple deployments, all using the same SQS URL. When a message is sent for Deployment 1, KEDA picks up the trigger and identifies which deployment the message corresponds to. It then only scales Deployment 1 by increasing its replica count accordingly.

asked 9 months ago110 views
1 Answer
0

Yes, it is possible to use a single SQS queue to trigger multiple deployments without running them simultaneously, but you'll need to implement a specific pattern to achieve this behavior.

When using KEDA with SQS for event-driven scaling, you can control which deployment scales based on message attributes or content. Here are a few approaches:

  1. Message Group IDs with FIFO Queue: If you use an SQS FIFO queue, you can assign different message group IDs to messages intended for different deployments. This ensures messages within a group are processed sequentially. With a batch size of 1, you can ensure only one message from the queue is consumed at a time.

  2. Message Filtering: You can include deployment identifiers in your messages and have your consumer logic determine which deployment should handle each message. This way, even though all deployments might be triggered by the same queue, only the appropriate one would actually process the message.

  3. Concurrency Control: Implement a locking mechanism using a service like DynamoDB to ensure only one deployment process runs at a time. When a deployment starts, it acquires a lock, and when it completes, it releases the lock and checks if there are more messages waiting.

  4. Step Functions with Queue: You could have a Lambda function check if a Step Function execution is already running when a new message arrives. If one is running, the message stays in the queue; if not, the Lambda triggers a new Step Function execution.

For the KEDA implementation specifically, you would need to configure your ScaledObjects with appropriate filtering or targeting logic to ensure they only scale the intended deployment based on message attributes or content.

Remember that with this approach, you'll need to carefully design your message structure and processing logic to avoid situations where messages get processed by the wrong deployment or get stuck in the queue.
Sources
Deployment Strategy SQS | AWS re:Post
How do I queue messages in SQS to wait for a Step Function to complete? | AWS re:Post
Set up event-driven auto scaling in Amazon EKS by using Amazon EKS Pod Identity and KEDA - AWS Prescriptive Guidance

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.