IoT Events Detector Model not receiving/logging all events, leading to premature timer expiry

Question

TLDR: The detector model appears to work as expected based on the received messages, the issue seems to be that not all of the events are being delivered to the detector.

I have used the CDK for Typescript and the @aws-cdk/aws-iotevents-alpha package to create an IoT Events Detector model to track whether devices are producing data.  The model has two states, Online and Offline, and an incoming event resets a model timer that covers the maximum period that data can be allowed to be missing (75 seconds).

The devices are actually sending multiple messages per minute (typically 6 in a test environment, production will use more relaxed timer and message rates).  Whilst the device messages are received on IoT Core mqtt, the messages are initially reformatted and published to EventBridge (as there are multiple different device payload formats to consider).  Messages are not evenly distributed across the minute (due to per message jitter), but the expectation is that a device will send at least one per minute and that the 75 second timeout is a reasonable compromise in case of occasional extended latency.

I then have an Eventbridge Rule that forwards the events to a lambda that uses the IoT Events client to put the messages to IoT events.  I can log this and see that messages are received multiple times per minute, and each results in a successful return from the client with no error entries.  Each lambda invocation effectively represents a single event to be put to IoT Events for consideration.

The model correctly creates a detector per device based on a field in the input data.

I have enabled IoT Event detector logging at Debug level, but this does not show all of the events.  It seems to show an event about once every minute, but with enough jitter that the 75 second timer maintained by the model can expire prematurely.  As a result the detector model toggles from Online to Offline periodically, despite there being events being sent to IoT events.  Even without logging, the behaviour is the same, so its note a case of the logging being only a subset of events - instead it definitely appears that only some events are received.

I've checked the quotas for IoT Events and message rates etc all seem well in excess of whats needed.  I've also checked latencies and none of these would explain the missing data points.

Here is the model in JSON form:

{
        "detectorModelDefinition": {
            "states": [
                {
                    "stateName": "Online",
                    "onInput": {
                        "events": [
                            {
                                "eventName": "input-event",
                                "condition": "currentInput(\"DevLorawanGatewayMonitorDetector_4aeeece1\")",
                                "actions": [
                                    {
                                        "resetTimer": {
                                            "timerName": "OnlineTimer"
                                        }
                                    },
                                    {
                                        "setVariable": {
                                            "variableName": "eventCount",
                                            "value": "$variable.eventCount + 1"
                                        }
                                    },
                                    {
                                        "setVariable": {
                                            "variableName": "lastSeen",
                                            "value": "$input.DevLorawanGatewayMonitorDetector_4aeeece1.time"
                                        }
                                    }
                                ]
                            }
                        ],
                        "transitionEvents": [
                            {
                                "eventName": "Online_to_Offline",
                                "condition": "timeout(\"OnlineTimer\")",
                                "actions": [],
                                "nextState": "Offline"
                            }
                        ]
                    },
                    "onEnter": {
                        "events": [
                            {
                                "eventName": "enter-event",
                                "condition": "currentInput(\"DevLorawanGatewayMonitorDetector_4aeeece1\")",
                                "actions": [
                                    {
                                        "setTimer": {
                                            "timerName": "OnlineTimer",
                                            "seconds": null,
                                            "durationExpression": "75"
                                        }
                                    },
                                    {
                                        "setVariable": {
                                            "variableName": "eventCount",
                                            "value": "1"
                                        }
                                    },
                                    {
                                        "setVariable": {
                                            "variableName": "lastSeen",
                                            "value": "$input.DevLorawanGatewayMonitorDetector_4aeeece1.time"
                                        }
                                    }
                                ]
                            }
                        ]
                    },
                    "onExit": {
                        "events": [
                            {
                                "eventName": "exit-event",
                                "condition": null,
                                "actions": [
                                    {
                                        "clearTimer": {
                                            "timerName": "OnlineTimer"
                                        }
                                    }
                                ]
                            }
                        ]
                    }
                },
                {
                    "stateName": "Offline",
                    "onInput": {
                        "events": [],
                        "transitionEvents": [
                            {
                                "eventName": "Offline_to_Online",
                                "condition": "currentInput(\"DevLorawanGatewayMonitorDetector_4aeeece1\")",
                                "actions": [],
                                "nextState": "Online"
                            }
                        ]
                    },
                    "onEnter": {
                        "events": []
                    },
                    "onExit": {
                        "events": []
                    }
                }
            ],
            "initialStateName": "Online"
        },
        "detectorModelDescription": "Detects online/offline state of a gateway",
        "detectorModelName": "DevLorawanGatewayMonitorDetector-4aeeece1",
        "evaluationMethod": "BATCH",
        "key": "eui",
        "roleArn": "arn:aws:iam::907963324750:role/Dev-LorawanGatewayMonitor-DetectorDetectorModelRole-A3USIpADUxoV"
    }

re:Post Agent · Answer

The issue you're experiencing with your IoT Events Detector Model not receiving or logging all events, leading to premature timer expiry, could be related to a few factors:

1. Batch Evaluation: Your detector model is using the "BATCH" evaluation method. This means that IoT Events processes inputs in batches, which can introduce some delay in processing and may not immediately reflect all incoming events. This could explain why you're seeing events approximately once per minute instead of for every message.

2. Input Throttling: While you've checked the quotas, it's worth noting that IoT Events has a default input throttling limit. If you're sending a high volume of messages, some might be throttled, leading to missed events.

3. Lambda Invocation: Since you're using a Lambda function to put messages into IoT Events, there could be potential issues with Lambda execution, such as timeouts or throttling, which might cause some events to be missed.

4. EventBridge to Lambda latency: The path from EventBridge to Lambda and then to IoT Events introduces additional points where messages could be delayed or lost.

To address these issues, you could try the following:

1. Change the evaluation method to "SERIAL" instead of "BATCH". This processes inputs as they arrive, which might be more suitable for your use case.

2. Increase the timer duration to account for potential delays in message processing.

3. Implement a retry mechanism in your Lambda function for putting messages to IoT Events.

4. Consider sending messages directly to IoT Events from IoT Core using IoT Rules, bypassing EventBridge and Lambda, which could reduce latency and potential points of failure.

5. Implement additional logging in your Lambda function to ensure all messages are being processed and successfully sent to IoT Events.

6. Monitor the IoT Events metrics in CloudWatch to check for throttling or other issues.

If the problem persists after trying these solutions, you may need to contact AWS support for further investigation, as there could be an underlying issue with the IoT Events service itself.
      
**Sources**  
[@aws-cdk/aws-iotevents-actions-alpha module · AWS CDK](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-iotevents-actions-alpha-readme.html)  
[Difficulty Implementing Basic Ingest Feature in AWS IoT Core | AWS re:Post](https://repost.aws/questions/QUHqyUoNzDQJ2QwbwlQfsCrg/difficulty-implementing-basic-ingest-feature-in-aws-iot-core)  
[class IotEventsPutMessageAction · AWS CDK](https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-iot-actions-alpha.IotEventsPutMessageAction.html)

Dave Meehan · Answer

TLDR; when using BatchPutMessageCommand with the IoTEventsDataClient, the messageId must be unique across requests, at least over a time of 1 minute, or the messages will be deduplicated.

The problem was down to the messageId included when putting events to IoT Events.  The documentation says that this should be unique within a batch submitted with batch put.  As a result, I had used a stringised version of the array index. IoT Events must then batch items together (the detector models evaluation method of BATCH was not a factor here, as I'd already tried SERIAL) over a certain period of time and deduplicated based on the messageId.  I have assumed as the rest of the input content would be unique based on the timestamp field, but it appears to deduplicate based on the messageId, over some period of time around 1 minute.

IoT Events Detector Model not receiving/logging all events, leading to premature timer expiry

Add your answer

Relevant content