Sending high throughput messages to IOT devices

0

We've been using a single backend app that connected via MQTT to listen for devices messages and send replies. After hitting the 100msg/sec limit we have switched to the HTTPS action to receive mqtt messages and we're still using a single MQTT connection to send replies.

However if the replies to be sent exceed the 100msg/sec limit of a single connection, what's the best way to go past that limit? We would like to avoid having to create multiple MQTT connections which seems to be just a workaround rather than a well thought solution.

Should we use the https data endpoint? Or this API?

alex88
asked a year ago252 views
2 Answers
2
Accepted Answer

Hi alex88,

to receive messages from your application you can use the rules engine to route messages to your application to overcome the 100msg/s quota for a single connection. You could route messages for example to Kinesis or SQS and then to your application.

How do you send messages to your devices? Do you send a message per device? You could think about using a single topic for your devices then you need to send only one message to address them all. Please see also the documentation for MQTT communication patterns.

In case you need to send a particular message to every device you could think about multiple MQTT connections or send publish requests in parallel.

Cheers,
Philipp

AWS
EXPERT
answered a year ago
profile picture
EXPERT
reviewed 23 days ago
  • Hi Philipp, regarding sending we're good now, using rules engine was the only way. We send responses to every request we receive so per device answers by using a per device topic root. When you say publish requests which API you are referring to?

  • Hi alex88, I meant this Publish API: https://docs.aws.amazon.com/iot/latest/apireference/API_iotdata_Publish.html. You can also use an MQTT connection but as you mentioned you have to work around the 100msg/s quota with multiple connection. The idea is to use the Publish API with multiple requests in parallel.

1

While HTTPS actions works to overcome the 100 msg/s limit, it is not the optimal solution unless your backend application runs outside of AWS. A better option, as Philipp mentioned, is to use an SQS or a Kinesis action. Kinesis is recommended if message ordering is significant for your processing, otherwise you should use SQS.

With HTTPS action you need to consider how to handle the following scenarios:

  1. traffic peaks that your backend is not able to process will result in lost messages
  2. if your backend is for some reason unavailable, messages will be lost

Using SQS, you decouple the messages from the application consuming it. For example, you could write your application code using Lambda functions, that would automatically scale to consume the messages in the queue. In case your application has bugs and cannot process the messages, the messages would still be retained in the queue, thus creating a more reliable system.

As for publishing from the backend to AWS IoT Core, you SHOULD use https://docs.aws.amazon.com/iot/latest/apireference/API_iotdata_Publish.html. Your caller need to have credentials with permissions to invoke this API which is best achieved by using an IAM Role. If you use Lambda functions that would be the function execution role.

You can check a reference architecture showing these principles here:

https://aws.amazon.com/solutions/implementations/smart-product-solution/

AWS
EXPERT
answered a year ago
  • Got it thank you, I'll switch to the Publish API

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions