StreamManager | Recover Persisted Data from File
I built a custom component that uses aws.greengrass.StreamManager component to export to a Kinesis stream.
I'm sending dummy helloaaaaaaaa like this:
self.stream_manager_client.append_message(self.modbus_data_stream_name, b'helloaaaaaaaaaaaaaa')
The message stream persists the messages in the file system in .index and .log files. Like so:
The content of the .log files looks like the following, which appears to be some "meta data" mixed with my message.
One use case we're trying to support is this - when the device loses internet, a technician can retrieve the device and regain the messages it was supposed to upload. I'm wondering if I can get support on how to decode the files in order to parse my messages and any meta data.
- 最新
- 投票最多
- 评论最多
Even when offline, you can use the read_messages API to read data from a stream from an arbitrary starting point. https://docs.aws.amazon.com/greengrass/v2/developerguide/work-with-streams.html#streammanagerclient-read-messages
Cheers,
Michael
This isn't documented because you aren't supposed to read the data manually, I absolutely will point you to the read messages API to do this for you.
With that said, the header format is 32 bytes total, ending with the payload length:
Thanks for suggesting the
read_messagesAPI. Would this API only work when called inside of a Greengrass component? Is there an API we can use without spinning up a Greengrass component?Normally you'd need a component yes just to authenticate with Stream manager, but you can also disable authentication in Stream manager.
STREAM_MANAGER_AUTHENTICATE_CLIENThttps://docs.aws.amazon.com/greengrass/v2/developerguide/stream-manager-component.html#stream-manager-component-configurationOk thanks for that pointer - I will verify that. I have another follow-up question: It appears to me that the StreamManager persists the data regardless of whether the data is successfully exported, what's the intention behind that design? An alternative in my mind is to remove the successfully exported data, which will save storage space in the core device.