By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

AWS IoT Core with binary paylad

0

We are planning on ingesting messages over MQTT using IoT Core, where the payload will be binary (more specifically, protobuf). Data will be forwarded using a rule to Kinesis. In the rule, we also want to add principal() to the message. Whether the message is still binary or not is not important.

We first looked into using the decode function for protobuf messages, as described here: https://docs.aws.amazon.com/iot/latest/developerguide/binary-payloads.html#binary-payloads-protobuf. However, this requires storing the protobuf .desc file in S3, and I noticed through cloudtrail that it makes frequent requests (maybe every request? Not sure) to S3, which would incur a high cost at our message load.

Next, we decided to just pass it on in binary through kinesis, and do the decoding in the kinesis consumer. However, in the documentation it's mentioned that "If you send a raw binary payload, AWS IoT Core routes it downstream to an Amazon S3 bucket through an S3 action. The raw binary payload is then encoded as base64 and attached to JSON.". Am I understanding it correctly that if we send a binary payload we cannot avoid incurring S3 charges? Would our only option be to base64 encode (or similar) the payload before sending it from our sensor device?

Topics
Internet of Things (IoT)Analytics
Tags
AWS IoT CoreAmazon Kinesis Data Streams
Language
English
FredrikJonsen
asked a year ago406 views
1 Answer
1
Accepted Answer

If you want to add additional data to the message before sending it to Kinesis Data Streams, you'll need to send it as a JSON payload. So your rule SQL will be similar to:

SELECT encode(*, 'base64') AS data, principal() AS principal FROM 'topic/subtopic'

In the Lambda function that is processing the stream, you'll need to base64 decode the data field to restore the binary payload and then decode the protobuf.

Amazon S3 is not involved in this flow, unless this is used by your Lambda function to retrieve the protobuf schema.

AWS
EXPERT
MassimilianoAWS
answered a year ago
  • FredrikJonsen
    a year ago

    Thank you, something like this is indeed what we had in mind. I have to ask though, was it a typo when you said "Amazon S3 is involved in this flow", and it actually should have said isn't, or is S3 actually involved in this? Assume you can ignore everything that happens after Kinesis, my question is only in regards to whether IoT Core itself will make calls to S3. To be even more precise, what I'm really looking for is clarification of the sentence "If you send a raw binary payload, AWS IoT Core routes it downstream to an Amazon S3 bucket through an S3 action." found in the documentation..

  • MassimilianoAWS EXPERT
    a year ago

    I meant to say "Amazon S3 is not involved" (corrected now). As for the documentation, that statement is utterly confusing. Thanks you for bringing to our attention. Binary payloads sent to AWS IoT Core are not written to S3 unless you use a specific S3 action in the rule. What I think that the documentation was supposed tos state is that to store a binary payload (eg a jpeg encoded image) to S3 you can use SELECT * from 'topic/subtopic and an S3 action.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content