I built a custom component that uses `aws.greengrass.StreamManager` component to export to a Kinesis stream. I'm sending dummy `helloaaaaaaaa` like this: ``` self.stream_manager_client.append_message(self.modbus_data_stream_name, b'helloaaaaaaaaaaaaaa') ``` The message stream persists the messages in the file system in `.index` and `.log` files. Like so: ![Enter image description here](/media/postImages/original/IMX7f_XVyyTuWeX3mtksh-uQ) The content of the `.log` files looks like the following, which appears to be some "meta data" mixed with my message. ![Enter image description here](/media/postImages/original/IM5bEeJ5lYQXaMd7kOLs2e2A) One use case we're trying to support is this - when the device loses internet, a technician can retrieve the device and regain the messages it was supposed to upload. I'm wondering if I can get support on how to decode the files in order to parse my messages and any meta data.

StreamManager | Recover Persisted Data from File

I built a custom component that uses aws.greengrass.StreamManager component to export to a Kinesis stream. I'm sending dummy helloaaaaaaaa like this:

self.stream_manager_client.append_message(self.modbus_data_stream_name, b'helloaaaaaaaaaaaaaa')

The message stream persists the messages in the file system in .index and .log files. Like so: Enter image description here

The content of the .log files looks like the following, which appears to be some "meta data" mixed with my message. Enter image description here

One use case we're trying to support is this - when the device loses internet, a technician can retrieve the device and regain the messages it was supposed to upload. I'm wondering if I can get support on how to decode the files in order to parse my messages and any meta data.

주제: 사물 인터넷(IoT)
태그: AWS IoT Greengrass
언어: English

ictwist

질문됨 4년 전387회 조회

1개 답변

최신
최다 투표
가장 많은 댓글

수락된 답변

Even when offline, you can use the read_messages API to read data from a stream from an arbitrary starting point. https://docs.aws.amazon.com/greengrass/v2/developerguide/work-with-streams.html#streammanagerclient-read-messages

Cheers,

Michael

전문가

MichaelDombrowski-AWS

답변함 4년 전

MichaelDombrowski-AWS 전문가

4년 전

This isn't documented because you aren't supposed to read the data manually, I absolutely will point you to the read messages API to do this for you.

With that said, the header format is 32 bytes total, ending with the payload length:

+-----------+---------+-----------+--------------+---------+-----------+---------------+
| magicSeq  | version | relSeqNum | bytePosition |  crc       | timestamp | payloadLength |
|  3 bytes     | 1 byte   |  4 bytes       |  4 bytes.       | 8 bytes |  8 bytes      |  4 bytes             |
+-----------+---------+-----------+--------------+---------+-----------+---------------+

ictwist
4년 전
Thanks for suggesting the read_messages API. Would this API only work when called inside of a Greengrass component? Is there an API we can use without spinning up a Greengrass component?
MichaelDombrowski-AWS 전문가
4년 전
Normally you'd need a component yes just to authenticate with Stream manager, but you can also disable authentication in Stream manager. STREAM_MANAGER_AUTHENTICATE_CLIENT https://docs.aws.amazon.com/greengrass/v2/developerguide/stream-manager-component.html#stream-manager-component-configuration
ictwist
4년 전
Ok thanks for that pointer - I will verify that. I have another follow-up question: It appears to me that the StreamManager persists the data regardless of whether the data is successfully exported, what's the intention behind that design? An alternative in my mind is to remove the successfully exported data, which will save storage space in the core device.

StreamManager | Recover Persisted Data from File

답변 추가

관련 콘텐츠