Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

StreamManager | Recover Persisted Data from File

0

I built a custom component that uses aws.greengrass.StreamManager component to export to a Kinesis stream. I'm sending dummy helloaaaaaaaa like this:

self.stream_manager_client.append_message(self.modbus_data_stream_name, b'helloaaaaaaaaaaaaaa')

The message stream persists the messages in the file system in .index and .log files. Like so: Enter image description here

The content of the .log files looks like the following, which appears to be some "meta data" mixed with my message. Enter image description here

One use case we're trying to support is this - when the device loses internet, a technician can retrieve the device and regain the messages it was supposed to upload. I'm wondering if I can get support on how to decode the files in order to parse my messages and any meta data.

Topics
Internet of Things (IoT)
Tags
AWS IoT Greengrass
Language
English

asked 4 years ago379 views

1 Answer
2
Accepted Answer

Even when offline, you can use the read_messages API to read data from a stream from an arbitrary starting point. https://docs.aws.amazon.com/greengrass/v2/developerguide/work-with-streams.html#streammanagerclient-read-messages

Cheers,

Michael

AWS
EXPERT

answered 4 years ago

  • MichaelDombrowski-AWS EXPERT
    4 years ago

    This isn't documented because you aren't supposed to read the data manually, I absolutely will point you to the read messages API to do this for you.

    With that said, the header format is 32 bytes total, ending with the payload length:

    +-----------+---------+-----------+--------------+---------+-----------+---------------+
| magicSeq  | version | relSeqNum | bytePosition |  crc       | timestamp | payloadLength |
|  3 bytes     | 1 byte   |  4 bytes       |  4 bytes.       | 8 bytes |  8 bytes      |  4 bytes             |
+-----------+---------+-----------+--------------+---------+-----------+---------------+
  • ictwist
    4 years ago

    Thanks for suggesting the read_messages API. Would this API only work when called inside of a Greengrass component? Is there an API we can use without spinning up a Greengrass component?

  • MichaelDombrowski-AWS EXPERT
    4 years ago

    Normally you'd need a component yes just to authenticate with Stream manager, but you can also disable authentication in Stream manager. STREAM_MANAGER_AUTHENTICATE_CLIENT https://docs.aws.amazon.com/greengrass/v2/developerguide/stream-manager-component.html#stream-manager-component-configuration

  • ictwist
    4 years ago

    Ok thanks for that pointer - I will verify that. I have another follow-up question: It appears to me that the StreamManager persists the data regardless of whether the data is successfully exported, what's the intention behind that design? An alternative in my mind is to remove the successfully exported data, which will save storage space in the core device.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content