Get Hands-on with Amazon EKS - Workshop Event Series
Whether you're taking your first steps with Kubernetes or you're an experienced practitioner looking to sharpen your skills, our Amazon EKS workshop series delivers practical, real-world experience that moves you forward. Learn directly from AWS solutions architects and EKS specialists through hands-on sessions designed to build your confidence with Kubernetes. Register now and start building with Amazon EKS!
Problems with Log manager component
Hello,
I have several Endpoints (EP) deployed in production working with the Greengrass framework. The architecture is very simple, one component is for receiving messages, and the other one is for showing messages on a screen. The components communicate through IPC. Also, I have deployed the Log Manager component, provided by AWS, for logging messages to Cloudwatch. The system has worked for 3 months with almost no problems (sometimes I have to restart the Greengrass daemon for the system to work correctly). This week one of the EP has not logged anything at all, although is connected to the internet, since I can ssh it. I have attached the log of the devices, can you help me to diagnose the problem?
2024-07-15T16:00:02.441Z [WARN] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
com.aws.greengrass.deployment.exceptions.AWSIotException: Unable to get response
at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:95)
at com.aws.greengrass.iot.IotCloudHelper.lambda$sendHttpRequest$1(IotCloudHelper.java:80)
at com.aws.greengrass.util.BaseRetryableAccessor.retry(BaseRetryableAccessor.java:32)
at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:81)
at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:206)
at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:327)
at com.aws.greengrass.tes.CredentialRequestHandler.getAwsCredentials(CredentialRequestHandler.java:336)
at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:24)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:88)
... 32 more
2024-07-15T16:00:02.445Z [ERROR] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials, credentialData=Failed to get connection}
2024-07-15T16:00:02.445Z [ERROR] (pool-3-thread-23) com.aws.greengrass.logmanager.CloudWatchLogsUploader: Unable to upload logs for log group /aws/greengrass/UserComponent/us-west-2/com.prod.screen. {}
software.amazon.awssdk.core.exception.SdkClientException: Failed to fetch credentials
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:31)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2024-07-15T16:00:03.663Z [WARN] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
com.aws.greengrass.deployment.exceptions.AWSIotException: Unable to get response
at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:95)
at com.aws.greengrass.iot.IotCloudHelper.lambda$sendHttpRequest$1(IotCloudHelper.java:80)
at com.aws.greengrass.util.BaseRetryableAccessor.retry(BaseRetryableAccessor.java:32)
at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:81)
at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:206)
at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:327)
at com.aws.greengrass.tes.CredentialRequestHandler.getAwsCredentials(CredentialRequestHandler.java:336)
at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:24)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
- Topics
- Internet of Things (IoT)
- Language
- English
- Newest
- Most votes
- Most comments
Hi. It appears to be a DNS or network issue, not a Greengrass issue:
Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com
Greengrass needs to refresh IAM credentials before Log Manager can write to CloudWatch.
Relevant content
AWS OFFICIALUpdated 2 years ago