Lambda handling AWS IoT traffic dropped a single message

0

I was looking into a bug recently where it seems like the lambda for processing MQTT messages from an IoT source just dropped a single message somehow. Our MQTT client (using the Java AWS IoT SDK) logged the message being sent successfully to MQTT (with a unique ID), but the logs in AWS from the lambda handling MQTT messages don't show anything with this ID, or any errors from around that time. Lots of other messages go through successfully around that time, it's just one that gets dropped.

I noticed that around the time the message should have been received in AWS, the lambda spun up two more execution environments to deal with load. The log group for one of these new instances shows it processing a bunch of messages successfully, but the log group for the other one doesn't have anything. I'm thinking maybe the message got sort of routed to the new instance getting spun up, but then that instance failed to actually start for some reason, and so the message got dropped?

I'm wondering if this explanation sounds plausible and is maybe an expected behavior for lambdas (and something we just have to work around in our application logic) or if I should be digging deeper into something that might have gont wrong in MQTT or in our SDK client.

zosnap
asked 10 days ago77 views
1 Answer
0

Usually Lambda will emit error messages in case of failures. I would look deeper into the client side.

I would also recommend add a Lambda Destination so that the case of failures, you can get notified about it.

profile pictureAWS
EXPERT
Uri
answered 9 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions