Handling the IoT connection failure scenarios using MQTT

Question

Hi, 
We are publishing the telemetry data to AWS IOT core every 1 sec from Raspberry Pi. We have a scenario where the connection failure due poor network or connection lost, the MQTT packets would be lost. We can use a queue mechanism for the lost data, and it resend to cloud when connection is restored. We have 2 questions:
1. How to handle both the live data and resending of lost data at device and cloud side. Our purpose on cloud to show live dashboarding as well as to run analytics for older data.
2. If connection issue is more than a day, how to manage the huge data (86440 records) and resend to cloud.

Just a note, we are not using any greengrass core for our application. Please suggest best way to handle these scenarios.

Answer

Hi. A single MQTT connection can support up to 100 publishes (sum of in an out) per second.

On the device side, one approach you can take is to maintain a message queue, as you said. Keep putting your new/live message at the head of the queue for immediate transmission and additionally publish up to 99 of the old messages per second. Potentially you can catch up on a day of messages in as little as 15 minutes. In practice it will be slower, because you have to be careful to not be throttled.

Do your messages contain a device timestamp and/or a sequence number? I think they will need to so that the cloud can distinguish live/new messages from old. That way only the new messages are directed to your live dashboarding.

Handling the IoT connection failure scenarios using MQTT

Relevant content