Hybrid Pub/Sub Solution Options

0

I wanted to check here to see if anyone had any better solution ideas for a problem we are attempting to solve for a client.

The client has on-premise services running which are generating messages. The end goal is: the messages come from their on-premise services, have a "current state" copy cached somewhere in AWS, and then get pushed out through an AWS websocket client.

The client has provided 3 options for getting the message to AWS:

  1. The messages sit in their on-premise Kafka and an AWS Service consumes messages from it
  2. The messages sit in their on-premise RabbitMQ and an AWS Service consumes messages from it
  3. A websocket is exposed on their on-premise service and an AWS Service subscribes to that websocket

We've done some research into several options for this.

  1. For the Kafka option, we are thinking setting up Lambda as a consumer to their on-premise Kafka (that is well documented). Lambda would just receive the message, save the "current state" in Dynamo or S3, and then publish through API Gateway for the websocket piece. The downside to this approach is that the messages are intended to retain their order, and using Lambda as a Kafka consumer doesn't exactly guarantee order. We can't tell if it's possible to have Lambda restrict itself to a single consumer, which would technically retain order, but this really feels like a situation where we aren't using the best tools for the job.
  2. We have not been able to hunt down many examples of consuming on-premise RabbitMQ from AWS Services. Current suspicions is that if we could find examples, it might have similar downsides to the Kafka approach.
  3. The websocket approach had us researching a handful of different things. We haven't really been able to find any services in AWS that can subscribe to that websocket and trigger an event when new messages come in. The best idea we could come up with here was spinning up a micro EC2 that subscribes to that websocket, receives messages, stores "current state" in Dynamo or S3, and then publishes through API Gateway for the websocket. In an ideal world, we would be accomplishing this in a way that better utilizes AWS Services and doesn't depend on a constantly running EC2. The websocket approach solves the message ordering issue, because we won't have to worry about old messages due to network connectivity issues or anything like that. On-premise has its own "current state", so if network connectivity issues happen, AWS would just end of getting the "current state" when it is able to reconnect, and all the messages in between would correctly never get to AWS.

We tried looking into IoT and EventBus because our research took us down those paths, but nothing there seems like it can really pull messages from the on-premise like we need it to.

We currently feel like options 1 and 3 are our best ideas right now. We wanted to quickly check with the community here in case someone was aware of anything that we hadn't considered yet.

asked a year ago190 views
1 Answer
0

I would suggest using the first option. i.e., place the events in Kafka and consume them with a Lambda function. Order of messages is preserved within each partition.

profile pictureAWS
EXPERT
Uri
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions