Skip to content

greengrass TokenExchangeService component fails to respond

0

We had a device running aws.greengrass.TokenExchangeService 2.0.3, and during a deployment upgrade it looks to have stopped responding to requests, although it was still in the "RUNNING" state. During this time greengrass had successfully connected and remained connected to IoT - however greengrass.log showed a CredentialRequestHandler error. Also, direct requests to TokenExchangeService failed. Note that the system remained in this error state for a number of days, never recovering. A system reboot recovered the issue.

Is there any reason why this should fail, is there a recommended monitoring and recovery process for something like thiks ?

root@nvidia:/data# curl  -s -H "Authorization: ${SVCUID}" ${AWS_CONTAINER_CREDENTIALS_FULL_URI}
Failed to get connection root@nvidia:/data#

greengrass.log

2025-12-11T22:01:31.752Z [WARN] (HTTP-Dispatcher) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/MyRoleAlias/credentials}
com.aws.greengrass.deployment.exceptions.AWSIotException: Unable to get response
        at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:95)
        at com.aws.greengrass.iot.IotCloudHelper.lambda$sendHttpRequest$1(IotCloudHelper.java:80)
        at com.aws.greengrass.util.BaseRetryableAccessor.retry(BaseRetryableAccessor.java:32)
        at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:81)
        at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:255)
        at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsWithTimeout(CredentialRequestHandler.java:223)
        at com.aws.greengrass.tes.CredentialRequestHandler.handle(CredentialRequestHandler.java:160)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
        at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:848)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:817)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$DefaultExecutor.execute(ServerImpl.java:201)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.handle(ServerImpl.java:560)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:526)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
  • TES component stays in RUNNING state because the local server that vends credentials is still up and running even when credential fetching from the iot credential service failed due to invalid device certificates at the time. Restarting Greengrass would not fix this until the TES role or the certificates are updated to valid ones.

asked a year ago186 views

2 Answers
0

The issue you're experiencing with the AWS IoT Greengrass Token Exchange Service (TES) component appears to be related to credential retrieval failures, despite the component showing as "RUNNING".

Based on the error logs, the token exchange service was unable to fetch credentials from AWS IoT Core. The specific error "Received fatal alert: bad_certificate" in the SSL handshake suggests there was an issue with the device's certificate when trying to authenticate with AWS IoT Core to retrieve credentials.

This can happen for several reasons:

  1. The device's certificate might have been temporarily invalid or expired
  2. There might have been network connectivity issues affecting the SSL handshake
  3. The role alias configuration might have changed during the deployment
  4. The AWS IoT credentials provider might have been temporarily unavailable

Greengrass core devices store AWS credentials in memory, and these credentials expire after an hour by default. If the token exchange service can't refresh these credentials, components that depend on AWS credentials will start failing, even though the token exchange service itself might still appear to be running.

For monitoring and recovery:

  1. Implement health checks that periodically test if the token exchange service is responding correctly (similar to your curl test)
  2. Set up automated recovery procedures that restart the token exchange service component if it fails health checks
  3. Consider implementing a watchdog that can trigger a system reboot if multiple recovery attempts fail
  4. Monitor the Greengrass logs for credential-related errors
  5. Ensure your device has a stable network connection to AWS IoT Core

Since a system reboot resolved your issue, it suggests that restarting the Greengrass services is an effective recovery mechanism. You could implement a less disruptive solution by creating a script that specifically restarts just the token exchange service component when it detects failures.
Sources
Troubleshooting AWS IoT Greengrass V2 - AWS IoT Greengrass
Token exchange service - AWS IoT Greengrass
Interact with AWS services - AWS IoT Greengrass

answered a year ago

0

The TokenExchangeService can appear “RUNNING” while failing to respond if its local IPC or role alias credential flow becomes stuck. This usually happens when the device loses a valid IoT Core MQTT session or when cached credentials expire but the service cannot refresh them through the role alias endpoint. Greengrass continues showing connected status because the nucleus maintains connectivity with IoT Core independently of TES, but the CredentialRequestHandler fails when IoT authentication breaks or the TES container’s network stack stalls. —Taz

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.