AWS IoT and Micro Services

Question

Hello, I am planning to use AWS IoT for my devices in the field to connect to my micro services in the cloud.   
Idea here is, the devices will publish to the topics that the micro-services will subscribe to. And the microservices will publish to the topics that the device subscribes to.   
  
So, for example, there are 2 devices d1 and d2. And 2 microservices ms1 and ms2.   
d1 will subscribe to devices/d1 topic  
d2 will subscribe to devices/d2 topic  
and,  
ms1 will subscribe to services/ms1 topic  
ms2 will subscribe to services/ms2 topic  
  
When d1 wants to send information to ms1, then d1 will publish on to services/ms1 topic and to send information to ms2, d1 will publish on to services/ms2 topic etc.   
Similarly, when ms1 wants to send information to d1, ms1 will publish on to devices/d1 topic etc  
I attached an image on how I visualize it.   
  
My concern here is, a micro service can have multiple instances running at the same time. So all the instances of ms1 will subscribe to services/ms1 topic. And when a device publishes a message on to that topic, how do we make sure only one instance to process that message and not multiple instances?  
I understand one of the ways to handle this is to use Lambda, where the message gets passed to lambda, and Lambda can call a REST API on the microservice. However, Lambda might have limitations to the scale I would like to achieve (100,000 devices with 1000 messages each every hour).   
  
So, wanted to see if there is a better way of doing it.  
  
Edited by: yarch on Nov 1, 2019 3:59 PM

Answer

You can use Rules Engine to do the 'services' topic subscriptions for you (create a Topic Rule that subscribe to 'services/ms1' and another rule for 'services/ms2'; or one rule that subscribes to 'services/+'). When a message is published to services/ms1 or services/ms2 then only the subscribed Rule(s) will fire (versus all subscribed micro-services).   
  
The rule can be used to perform a Lambda Action or HTTP Action. If your services have an HTTPS endpoint, then HTTP Action would be my preference to get over the Lambda concurrency limitation. For more information on HTTP Action see https://aws.amazon.com/blogs/iot/route-data-directly-from-iot-core-to-your-web-services/  
  
To make sure only 1 micro-service fires to handle the request, you could use a load balancer (https://aws.amazon.com/elasticloadbalancing/) or target specific instance from the Rule.

Answer

Hi yarch,  
  
I recommend checking out the Well-Architected IoT Lens whitepaper: https://d1.awsstatic.com/whitepapers/architecture/AWS-IoT-Lens.pdf This might answer some of your current and future questions with some guidance on how to think about architecting IoT solutions on AWS.  
  
In addition to Artem's advice, I would recommend decoupling your devices from the microservices by using message queues. For example, devices report their state or make a query to a topic oriented around itself (like devices/d1/reported or devices/d1/query) and then using the rules engine to contextualize which message queues a given message should be added to. SQS is a great tool for designing application workloads where requests are processed by a single worker. Your microservice worker would then read messages off the queue, execute logic, then send a message back to IoT Core. It is more acceptable for the microservice to know about topic architecture and could publish to a topic like devices/d1/response. This kind of design simplifies your devices (they just know about themselves), makes your architecture more agile (cloud structure can evolve without needing an update on the device), and achieves the singular processing of device messages in your microservice architecture.   
  
Hope this helps,  
Ryan

AWS IoT and Micro Services

Relevant content