Calculating measure from multi part LoRaWAN uplink message before storing in Timestream database

Question

Hi,

I'm working on an application in which I store data from LoRaWAN IoT sensors into an Amazon Timestream table. So far I've been able to decode and load data from "simple" sensors (devices that send all their measurements in a single message) by using Lambda (for binary decoding) and IoT Core's rule engine (for storing into Timestream).

The issue I'm encountering now is that I have a sensor that sends its data over two consecutive messages (Netvox R718N37, a current meter). Although the binary decoder can decode each individual message, I need to combine the data from both messages in order to build a record to load into the database.

Just for clarity, the example below corresponds to two messages received from the sensor

**Message 1:**
```
{
	"WirelessDeviceId": "aaabbbccc",
	"WirelessMetadata": {
		"LoRaWAN": {
			"ADR": true,
			"Bandwidth": 125,
			"ClassB": false,
			"CodeRate": "4/5",
			"DataRate": "3",
			"DevAddr": "123456789",
			"DevEui": "ggghhhiii",
			"FCnt": 203,
			"FOptLen": 0,
			"FPort": 6,
			"Frequency": "905300000",
			"Gateways": [{ "GatewayEui": "dddeeefff", "Rssi": -87, "Snr": 7.5 }],
			"MIC": "418bcec7",
			"MType": "UnconfirmedDataUp",
			"Major": "LoRaWANR1",
			"Modulation": "LORA",
			"PolarizationInversion": false,
			"SpreadingFactor": 7,
			"Timestamp": "2022-08-30T19:11:19Z"
		}
	},
	"PayloadData": "AUoBJAAAAAAAAAE=",
	"DecodedPayloadData": {
		"devicetype": "R718N3-1",
		"battery": 3.6,
		"currentma1": 0,
		"currentma2": 0,
		"currentma3": 0,
		"multiplier1": 1
	},
	"ArrivalTimestamp": 1661886679775
}
```

**Message #2:**

```
{
	"WirelessDeviceId": "aaabbbccc",
	"WirelessMetadata": {
		"LoRaWAN": {
			"ADR": true,
			"Bandwidth": 125,
			"ClassB": false,
			"CodeRate": "4/5",
			"DataRate": "3",
			"DevAddr": "123456789",
			"DevEui": "ggghhhiii",
			"FCnt": 204,
			"FOptLen": 0,
			"FPort": 6,
			"Frequency": "904100000",
			"Gateways": [{ "GatewayEui": "dddeeefff", "Rssi": -87, "Snr": 9.5 }],
			"MIC": "5a21d54e",
			"MType": "UnconfirmedDataUp",
			"Major": "LoRaWANR1",
			"Modulation": "LORA",
			"PolarizationInversion": false,
			"SpreadingFactor": 7,
			"Timestamp": "2022-08-30T19:11:29Z"
		}
	},
	"PayloadData": "AUoCJAEBAAAAAAA=",
  "DecodedPayloadData": {
    "devicetype": "R718N3-2",
    "battery": 3.6,
    "multiplier2": 1,
    "multiplier3": 1
	},
	"ArrivalTimestamp": 1661886689459
}
```

For instance, to calculate the current for phase 2 (a measurement I need in the Timestream table), I need to multiply `DecodedPayloadData.currentma2` and `DecodedPayloadData.multiplier2`, which are in different messages.

My question is: what strategy would you recommend for me to combine the two consecutive MQTT messages? I need a way to get both messages into a Lambda function so I can run the calculations and store the data into the database.

Thank you in advance,

Accepted Answer

Hi. It looks like the pair of messages come 10s apart, but how often is a pair sent by each device? How big is your fleet? Do you plan to use shadows already?

Sometimes if can be handy, from the LoRaWAN Lambda decoder, to store the device state in a Thing shadow. Especially if you want to use [fleet indexing](https://docs.aws.amazon.com/iot/latest/developerguide/iot-indexing.html) with your fleet at all. However, you have to be wary of shadow [costs](https://aws.amazon.com/iot-core/pricing/) and [limits](https://docs.aws.amazon.com/general/latest/gr/iot-core.html#device-shadow-limits). Especially if you have a large fleet and/or frequent uplinks.

This is the long way of saying that I think shadows could be a good solution. Since the uplinks are unconfirmed, you will frequently face the situation of missing one of the two messages in the pair. So it's good to manage reconstitution in some per-device way. And since it's only 2 messages, a queue is probably not the best fit. From the Lambda, you can identify if it's the first or second message. If it's the first, store it in the shadow. If it's the second, read the shadow, compare timestamps to make sure you have a pair, and combine the messages. Possibly store the combined result back in the shadow, as well as send it on to Timestream.

- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iot-data.html#IoTDataPlane.Client.update_thing_shadow
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iot-data.html#IoTDataPlane.Client.get_thing_shadow

With the data in the shadow (even if just the first message), you could perform [fleet indexing queries](https://docs.aws.amazon.com/iot/latest/developerguide/example-queries.html) such as "give me all the devices with battery below 3V".

If you are not interested in shadows or fleet indexing, I would suggest DynamoDB and use it similarly.

Calculating measure from multi part LoRaWAN uplink message before storing in Timestream database

Relevant content