I built a custom component that uses `aws.greengrass.StreamManager` component to export to a Kinesis stream. I'm sending dummy `helloaaaaaaaa` like this: ``` self.stream_manager_client.append_message(self.modbus_data_stream_name, b'helloaaaaaaaaaaaaaa') ``` The message stream persists the messages in the file system in `.index` and `.log` files. Like so: ![Enter image description here](/media/postImages/original/IMX7f_XVyyTuWeX3mtksh-uQ) The content of the `.log` files looks like the following, which appears to be some "meta data" mixed with my message. ![Enter image description here](/media/postImages/original/IM5bEeJ5lYQXaMd7kOLs2e2A) One use case we're trying to support is this - when the device loses internet, a technician can retrieve the device and regain the messages it was supposed to upload. I'm wondering if I can get support on how to decode the files in order to parse my messages and any meta data.

StreamManager | Recover Persisted Data from File

I built a custom component that uses aws.greengrass.StreamManager component to export to a Kinesis stream. I'm sending dummy helloaaaaaaaa like this:

self.stream_manager_client.append_message(self.modbus_data_stream_name, b'helloaaaaaaaaaaaaaa')

The message stream persists the messages in the file system in .index and .log files. Like so: Enter image description here

The content of the .log files looks like the following, which appears to be some "meta data" mixed with my message. Enter image description here

One use case we're trying to support is this - when the device loses internet, a technician can retrieve the device and regain the messages it was supposed to upload. I'm wondering if I can get support on how to decode the files in order to parse my messages and any meta data.

トピック: モノのインターネット (IoT)
タグ: AWS IoT Greengrass
言語: English

ictwist

質問済み 4年前388ビュー

1回答

新しい順
投票が多い順
コメントが多い順

承認された回答

Even when offline, you can use the read_messages API to read data from a stream from an arbitrary starting point. https://docs.aws.amazon.com/greengrass/v2/developerguide/work-with-streams.html#streammanagerclient-read-messages

Cheers,

Michael

エキスパート

MichaelDombrowski-AWS

回答済み 4年前

MichaelDombrowski-AWS エキスパート

4年前

This isn't documented because you aren't supposed to read the data manually, I absolutely will point you to the read messages API to do this for you.

With that said, the header format is 32 bytes total, ending with the payload length:

+-----------+---------+-----------+--------------+---------+-----------+---------------+
| magicSeq  | version | relSeqNum | bytePosition |  crc       | timestamp | payloadLength |
|  3 bytes     | 1 byte   |  4 bytes       |  4 bytes.       | 8 bytes |  8 bytes      |  4 bytes             |
+-----------+---------+-----------+--------------+---------+-----------+---------------+

ictwist
4年前
Thanks for suggesting the read_messages API. Would this API only work when called inside of a Greengrass component? Is there an API we can use without spinning up a Greengrass component?
MichaelDombrowski-AWS エキスパート
4年前
Normally you'd need a component yes just to authenticate with Stream manager, but you can also disable authentication in Stream manager. STREAM_MANAGER_AUTHENTICATE_CLIENT https://docs.aws.amazon.com/greengrass/v2/developerguide/stream-manager-component.html#stream-manager-component-configuration
ictwist
4年前
Ok thanks for that pointer - I will verify that. I have another follow-up question: It appears to me that the StreamManager persists the data regardless of whether the data is successfully exported, what's the intention behind that design? An alternative in my mind is to remove the successfully exported data, which will save storage space in the core device.

StreamManager | Recover Persisted Data from File

回答を追加する

関連するコンテンツ