Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Greengrass StreamManager java.net.UnknownHostException: kinesis.eu-west-1.amazonaws.com

0

Hello, we have a problem where the OS and other applications are able to normally resolve the kinesis endpoint, but StreamManager just get stuck with UnknownHostException and is unable to upload data. However this problem seems to be somewhat intermittent since it can upload some of it, just very little.

If it helps to add context, the devices experiencing these issues are connected via VSAT and they have very high latency and rather low (~1Mbps) throughput. Still, I would expect the error to be about something else. The connection is also very stable. It may be slow, but its on 24/7.

It has accumulated already 2.4GB of events over the past 3 months and it keeps accumulating. The data is not ending up to the cloud and customers are complaining about the unreasonably long delay.

Log exceprt (I ran it through jq for your convenience)

2025 Aug 01 19:30:52,876 ESC[1;31m[ERROR]ESC[m (pool-7-thread-2) com.amazonaws.iot.greengrass.streammanager.export.destination.Kin
esisDestination: Encountered error while exporting data to AWS Kinesis
com.amazonaws.SdkClientException: Unable to execute HTTP request: kinesis.eu-west-1.amazonaws.com
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:3434) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:3401) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:3390) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.services.kinesis.AmazonKinesisClient.executePutRecords(AmazonKinesisClient.java:2552) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.services.kinesis.AmazonKinesisClient.putRecords(AmazonKinesisClient.java:2521) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.iot.greengrass.streammanager.export.destination.KinesisDestination.sendMessage(KinesisDestination.java:78) ~[AWSG
reengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.iot.greengrass.streammanager.export.upload.MessageUploaderTask.upload(MessageUploaderTask.java:77) ~[AWSGreengras
sGreenlake-1.0-super.jar:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.UnknownHostException: kinesis.eu-west-1.amazonaws.com
at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1386) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1307) ~[?:?]
at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:113) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.conn.$Proxy22.connect(Unknown Source) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[AWSGreengrassGreenlake-1.0-super.jar:?]
... 18 more

The log seems to suggest that the exception comes from some kind of cache, however I did flush the OS caches and restarted the component using greengrass cli - same result. Besides - the cache should have timed out in 3 months and if it didn't, then no messages should have been sent. But it is sending some of them. I am unable to determine any patterns when the messages are sent and when they are not. Seems to be random.

This is the code we use for stream manager (Typescript)

for (const config of Object.values(this.config)) {
      const kinesisConfig = new KinesisConfig(
        config.streamName,
        config.streamName,
        config.batchSize,
        config.batchIntervalMillis,
        config.priority, // note the start sequence number is not specified since nobody knows what that actually does or what the default of 0 means
      );

      const streamDefinition = new MessageStreamDefinition(
        config.streamName,
        config.kinesisStreamMaxSizeBytes,
        config.kinesisStreamSegmentSizeBytes,
        null, // TTL, keep old data
        StrategyOnFull.OverwriteOldestData,
        Persistence.File,
        true,
        new ExportDefinition(null, null, [kinesisConfig]),
      );
      try {
        // streamNames is result of this.client.listStreams()
        if (streamNames.includes(config.streamName)) {
          logger.debug(`Updating message stream ${config.streamName}`);
          await this.client.updateMessageStream(streamDefinition);
          logger.debug(`Message stream ${config.streamName} updated!`);
        } else {
          logger.debug(`Creating message stream ${config.streamName}`);
          await this.client.createMessageStream(streamDefinition);
          logger.debug(`Message stream ${config.streamName} created!`);
        }
      } catch (err) {
        logger.error(`Could not create stream manager stream: ${err}`);
      }
}

Side question - if we recover the connection, is Stream manager going to retry forever and eventually flush all those old messages (they go all the way back to April 2025)? We have disabled ttl on streams and max size is 10GB. We also set the overwrite strategy to overwrite old.

  • Stanislav
    a year ago

    So I just tried it on a device in a location that has good internet and the StreamManager is just sitting there doing absolutely nothing. Not trying to flush or anything. The logs are just full of "emitting telemetry v2" messages and its not trying to do anything with the backed up events. How do we fix this?

  • Saranya
    a year ago

    Please provide the Stream manager version to see if the data upload is paused because of some other reason. It is possible that Stream manager JVM application's DNS cache is outdated but that should be fixed if you restart the greengrass or the stream manager component. I see that you tried to restart the component but it not clear if that is stream manager. Alternatively, if it is application's DNS that is actually causing the issue, you can update the JVM_ARGS configuration of the component to invalid the DNS cache with some TTL.

  • Stanislav
    a year ago

    I have tried restarting both the component and greengrass. I also tried restarting the whole computer and in 3 months definitely it was restarted plenty of times. That did not help. I am also more concerned for the fact that when there is good internet connectivity, the stream manager is not flushing the accumulated messages anyway. Its just sitting there doing nothing. Why is that? Does that have something to do with us not setting the sequence number in stream definitions? Versions of SM that I have tried are 2.2.0, 2.2.1 and 2.1.13

Topics
AnalyticsInternet of Things (IoT)
Tags
Amazon KinesisAWS IoT Greengrass
Language
English

asked a year ago127 views

1 Answer
-1

The issue you're experiencing with AWS IoT Greengrass StreamManager appears to be related to DNS resolution problems in a high-latency, low-bandwidth VSAT connection environment. The java.net.UnknownHostException indicates that the StreamManager component is unable to resolve the Kinesis endpoint domain name (kinesis.eu-west-1.amazonaws.com).

This intermittent behavior is likely due to DNS resolution timeouts occurring because of the high latency in your VSAT connection. While the OS might be able to resolve the domain in some cases, the Java DNS resolver used by StreamManager might be timing out before receiving a response from the DNS server.

Some potential solutions to consider:

  1. Check your DNS configuration and ensure it's properly set up for the high-latency environment.
  2. Consider implementing a local DNS caching solution to reduce the need for frequent DNS lookups.
  3. Adjust the JVM DNS caching parameters for the Greengrass component to increase the TTL (Time To Live) for DNS entries.
  4. If possible, configure a more reliable DNS server that can handle the high-latency requests better.

Regarding your side question: Yes, StreamManager is designed to persist data locally and retry sending it to AWS Cloud destinations. When connectivity is restored, StreamManager should attempt to send all the accumulated data based on the stream's configuration. The data is stored according to the stream's persistence and size settings. By default, StreamManager stores data in files on the core device's file system.

However, there are a few important considerations:

  • Make sure your stream is configured with sufficient storage capacity to handle the 2.4GB of accumulated data.
  • Check the stream's TTL (Time To Live) settings. If a TTL is set, messages older than that value might be discarded.
  • The stream's "Strategy on Full" setting determines what happens when the stream reaches its size limit (reject new data or overwrite old data).

You might want to check your stream configuration to ensure it's set up to retain data for the required period and has sufficient storage capacity to handle the backlog.
Sources
Use StreamManagerClient to work with streams - AWS IoT Greengrass
Export configurations for supported AWS Cloud destinations - AWS IoT Greengrass
Export configurations for supported AWS Cloud destinations - AWS IoT Greengrass
data-streams

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content