Skip to content

[AWS IoT Core] Stream Manager provided component stops putting records to Kinesis Data Stream

0

Since installing the latest version of Stream Manager 2.2.1, we have noticed that data is not being sent to Kinesis Data Stream. In some cases, putRecords requests are not even being made. However, when using the old version 2.1.12 on over 15 VMs, everything works correctly. Is it possible to find out the reason for this regression? Where can I view the changes to the Stream Manager code?

Env Info : RHEL 8, Java 11, Py 3.11

  • The Stream Manager component isn’t open-source, so there isn’t a public code repo to diff, but AWS does publish a detailed changelog in the Greengrass v2 docs. Version 2.2.1 actually includes a fix for a bug where Stream Manager failed to export messages to Kinesis Data Streams and adds performance improvements, while 2.1.12 mainly changes credential precedence. The recommended way to track down your regression is to check the aws.greengrass.StreamManager.log on an affected VM (optionally with LOG_LEVEL=DEBUG), and verify your Greengrass nucleus version and token-exchange IAM role. If the log shows export failures or auth issues, that will point to the exact cause. If everything looks correct but Kinesis calls still aren’t made, I’d open an AWS Support case and include the Stream Manager logs plus your component version/recipe, because 2.2.1 is expected to improve, not break, exports to Kinesis.

  • To add onto Eric's comment, the recent changes to the Kinesis Data Streams export have mainly to do with improving how the record allocation to shards works to avoid some throttling issues from previous versions. Are you observing that Stream Manager is not sending any data at all, or maybe is it falling behind in throughput and getting behind on data but data is still being uploaded?

1 Answer
0

The issue you're experiencing with Stream Manager v2.2.1 not sending data to Kinesis Data Stream is concerning, especially since v2.1.2 works correctly on your VMs.

Interestingly, according to the AWS IoT Greengrass documentation history, Stream Manager v2.2.1 was specifically released to fix an issue where "stream manager fails to export messages to Kinesis destinations" and to improve "the performance of stream manager exports to Kinesis destinations." This release was made available on March 12, 2025.

Since you're experiencing the exact issue that this version was meant to fix, there could be a few possibilities:

  1. There might be compatibility issues with your specific environment or configuration
  2. The fix might not address all scenarios where the export fails
  3. There could be a new bug introduced in v2.2.1

Stream Manager can sometimes encounter issues with sequence numbers being out of bounds, especially after unexpected shutdowns like VM reboots. This typically manifests as an "InvalidStreamPositionException: the sequence number is out of bounds" error, which can prevent Stream Manager from starting correctly.

For troubleshooting, I would recommend:

  1. Check the Stream Manager logs for specific error messages
  2. Verify that the Stream Manager component has the correct permissions to access your Kinesis Data Stream
  3. Ensure that your Kinesis Data Stream is properly configured and has sufficient capacity
  4. Consider temporarily reverting to v2.1.2 if it's critical for your operations

If you need to see the changes in Stream Manager code, AWS doesn't typically publish the internal code changes for their managed components. However, you can review the documentation history and release notes for high-level information about what was changed between versions.
Sources
[AWS IoT] Stream Manager error while Exporting data from local_streams | AWS re:Post
Document history for the AWS IoT Greengrass V2 Developer Guide - AWS IoT Greengrass

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.