Hybrid Pub/Sub Solution Options

0

I wanted to check here to see if anyone had any better solution ideas for a problem we are attempting to solve for a client.

The client has on-premise services running which are generating messages. The end goal is: the messages come from their on-premise services, have a "current state" copy cached somewhere in AWS, and then get pushed out through an AWS websocket client.

The client has provided 3 options for getting the message to AWS:

  1. The messages sit in their on-premise Kafka and an AWS Service consumes messages from it
  2. The messages sit in their on-premise RabbitMQ and an AWS Service consumes messages from it
  3. A websocket is exposed on their on-premise service and an AWS Service subscribes to that websocket

We've done some research into several options for this.

  1. For the Kafka option, we are thinking setting up Lambda as a consumer to their on-premise Kafka (that is well documented). Lambda would just receive the message, save the "current state" in Dynamo or S3, and then publish through API Gateway for the websocket piece. The downside to this approach is that the messages are intended to retain their order, and using Lambda as a Kafka consumer doesn't exactly guarantee order. We can't tell if it's possible to have Lambda restrict itself to a single consumer, which would technically retain order, but this really feels like a situation where we aren't using the best tools for the job.
  2. We have not been able to hunt down many examples of consuming on-premise RabbitMQ from AWS Services. Current suspicions is that if we could find examples, it might have similar downsides to the Kafka approach.
  3. The websocket approach had us researching a handful of different things. We haven't really been able to find any services in AWS that can subscribe to that websocket and trigger an event when new messages come in. The best idea we could come up with here was spinning up a micro EC2 that subscribes to that websocket, receives messages, stores "current state" in Dynamo or S3, and then publishes through API Gateway for the websocket. In an ideal world, we would be accomplishing this in a way that better utilizes AWS Services and doesn't depend on a constantly running EC2. The websocket approach solves the message ordering issue, because we won't have to worry about old messages due to network connectivity issues or anything like that. On-premise has its own "current state", so if network connectivity issues happen, AWS would just end of getting the "current state" when it is able to reconnect, and all the messages in between would correctly never get to AWS.

We tried looking into IoT and EventBus because our research took us down those paths, but nothing there seems like it can really pull messages from the on-premise like we need it to.

We currently feel like options 1 and 3 are our best ideas right now. We wanted to quickly check with the community here in case someone was aware of anything that we hadn't considered yet.

posta un anno fa211 visualizzazioni
1 Risposta
0

I would suggest using the first option. i.e., place the events in Kafka and consume them with a Lambda function. Order of messages is preserved within each partition.

profile pictureAWS
ESPERTO
Uri
con risposta un anno fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande