Token Exchange Service (TES) failing to fetch credentials

0

Gentlefolks, why is the the token exchange service (TES) failing to fetch credentials? I have greengrass installed on Ubuntu 20.x.x running on a virtual machine. At the end of the numbered items is an error log (truncated) obtained from /greengrass/v2/logs/greengrass.log. Thank you

What I've done or what I think you should know:

  1. There exist GreenGrassServiceRole which contains AWSGreengrassResourceAccessRolePolicy, and every other policy containing the word "greengrass" in its name. It also has a trust relationship as below. This was created when I installed Greengrass, but I added additional policies.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "greengrass.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "<MY_ACCOUNT_NUMBER>"
                },
                "ArnLike": {
                    "aws:SourceArn": "<ARN_THAT_SHOWS_MY_REGION_AND_ACC_NUMBER>:*"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "credentials.iot.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
  1. aws.greengrass.Nucleus component is deployed with the following configuration update. The alias also exists.
{
  "reset": [],
  "merge": {
    "ComponentConfiguration": {
      "DefaultConfiguration": {
        "iotRoleAlias": "GreengrassV2TokenExchangeRoleAlias",
        "awsRegion": "us-east-1",
        "iotCredEndpoint": "https://sts.us-east-1.amazonaws.com",
        "iotDataEndpoint": "<ENDPOINT_OBTAINED_FROM_IOT_CORE_SETTINGS>"
      }
    }
  }
}
  1. aws.greengrass.TokenExchangeService is deployed.

  2. There's a custom component that uses the Greengrass SDK to publish to IoT Core. It has the following configuration update.

{
  "reset": [],
  "merge": {
    "ComponentDependencies": {
      "aws.greengrass.TokenExchangeService": {
        "VersionRequirement": "^2.0.0",
        "DependencyType": "HARD"
      }
    }
  }
}
  1. There is an IoT policy from a previous exercise which attached to the core device (Ubuntu on virtual machine) certificate. It allows all actions. There's also another GreengrassTESCertificatePolicyGreengrassV2TokenExchangeRoleAlias which is associated with the thing's certificate. The policy allows iot:AssumeRoleWithCertificate

ERROR LOG BELOW

2022-05-21T21:07:39.592Z [ERROR] (pool-2-thread-28) com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials, credentialData=Failed to get connection}
2022-05-21T21:08:38.071Z [WARN] (pool-2-thread-28) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
com.aws.greengrass.deployment.exceptions.AWSIotException: Unable to get response
	at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:95)
	at com.aws.greengrass.iot.IotCloudHelper.lambda$sendHttpRequest$1(IotCloudHelper.java:80)
	at com.aws.greengrass.util.BaseRetryableAccessor.retry(BaseRetryableAccessor.java:32)
	at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:81)
	at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:207)
	at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:328)
	at com.aws.greengrass.tes.CredentialRequestHandler.getAwsCredentials(CredentialRequestHandler.java:337)
	at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:24)
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.resolveCredentials(AwsExecutionContextBuilder.java:165)
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:102)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
	at software.amazon.awssdk.services.s3.DefaultS3Client.getBucketLocation(DefaultS3Client.java:3382)
	at com.aws.greengrass.componentmanager.builtins.S3Downloader.lambda$getRegionClientForBucket$2(S3Downloader.java:134)
	at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:50)
	at com.aws.greengrass.componentmanager.builtins.S3Downloader.getRegionClientForBucket(S3Downloader.java:133)
	at com.aws.greengrass.componentmanager.builtins.S3Downloader.getDownloadSize(S3Downloader.java:115)
	at com.aws.greengrass.componentmanager.ComponentManager.prepareArtifacts(ComponentManager.java:420)
	at com.aws.greengrass.componentmanager.ComponentManager.preparePackage(ComponentManager.java:377)
	at com.aws.greengrass.componentmanager.ComponentManager.lambda$preparePackages$1(ComponentManager.java:338)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.UnknownHostException: https
	at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
	at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at jdk.internal.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:80)
	at com.sun.proxy.$Proxy15.connect(Unknown Source)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:253)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:106)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:232)
	at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:88)
	... 27 more

2022-05-21T21:08:38.073Z [ERROR] (pool-2-thread-28) com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials, credentialData=Failed to get connection}
asked 2 years ago1345 views
2 Answers
2
Accepted Answer

Hello,

The error is java.net.UnknownHostException: https. This indicates to me that the credential endpoint is not correctly configured. I see in your configuration that you are trying to use STS as the credential endpoint and this is not correct to setup TES. The AWS IoT Credential endpoint is the only endpoint you may use.

See: https://docs.aws.amazon.com/iot/latest/developerguide/authorizing-direct-aws.html to learn more about this endpoint. Find the correct endpoint for your account by running aws iot describe-endpoint --endpoint-type iot:CredentialProvider.

If you install Greengrass with automatic provisioning, all of this will be setup correctly for you. https://docs.aws.amazon.com/greengrass/v2/developerguide/quick-installation.html

The IoT Data and Credential endpoints which you configure Greengrass to use should only be the hostname, excluding any protocol (ex. https://). An example of a correct value would be xxxxxx.credentials.iot.us-east-1.amazonaws.com

AWS
EXPERT
answered 2 years ago
  • Unfortunately, it wouldn't budge. Thank you for the relatively quick response and provided options. Following the links you provided, I can confirm that I did install with automatic provisioning.

    What I did or what I think you should know after your last reply

    1. Adjusted the Nucleus configuration update to use AWS IoT endpoint gotten from using the "aws iot describe-endpoint..." utility.
    {
      "reset": [],
      "merge": {
        "ComponentConfiguration": {
          "DefaultConfiguration": {
            "iotRoleAlias": "GreengrassV2TokenExchangeRoleAlias",
            "awsRegion": "us-east-1",
            "iotCredEndpoint": "xxxxxxxxxxxxxxx.credentials.iot.us-east-1.amazonaws.com",
            "iotDataEndpoint": "yyyyyyyyyyyyyy-zzz.iot.us-east-1.amazonaws.com"
          }
        }
      }
    }
    
    1. Confirmed the trust policy pointed to by GreengrassV2TokenExchangeRoleAlias has 'credentials.iot.amazonaws.com'.

    I don't know what else to try.

  • This is the python script that should should oublish to IoT Core

        ipc_client = awsiot.greengrasscoreipc.connect()
    
        while True:
        	# _, _, free_m = map(int, os.popen('free -t -m').readlines()[-1].split()[1:])
    
            telemetry_data = {
                "ram": int(uniform(70,100)),
                "soc": open("/sys/class/power_supply/BAT0/status","r").readline().strip(),
                "tnw": int(round(time.time() * 1000))
            }
            
            op = ipc_client.new_publish_to_iot_core()
            op.activate(model.PublishToIoTCoreRequest(
                topic_name="<TOPIC_NAME>",
                qos=model.QOS.AT_LEAST_ONCE,
                payload=json.dumps(telemetry_data).encode(),
            ))
            try:
                result = op.get_response().result(timeout=5.0)
                print("successfully published message:", result)
            except Exception as e:
                print("failed to publish message:", e)
    
            time.sleep(5)
    
  • OK, I see that you are not using MERGE correctly. MERGE will change the configuration options using a map. You should not have ComponentConfiguration or DefaultConfiguration as part of MERGE.

    So it should look like:

    {"MERGE": {
      "iotRoleAlias": "GreengrassV2TokenExchangeRoleAlias",
       "awsRegion": "us-east-1",
       "iotCredEndpoint": "xxxxxxxxxxxxxxx.credentials.iot.us-east-1.amazonaws.com",
       "iotDataEndpoint": "yyyyyyyyyyyyyy-zzz.iot.us-east-1.amazonaws.com"
    }}
    

    Read more here: https://docs.aws.amazon.com/greengrass/v2/developerguide/update-component-configurations.html#merge-configuration-update

0

Thank you for the updates. I've successfully been able to publish to Iot Core

How I moved on after your updates

  1. Reset the updates and used merge correctly.
  2. Added the aws.greengrass.clientdevices.mqtt.Bridge and defined a mapping.
{
"mqttTopicMapping": {
      "map1": {
        "topic": "<MY_TOPIC>",
        "source": "LocalMqtt",
        "target": "IotCore"
      },
      "map2": {
        "topic": "MY_TOPIC",
        "source": "Pubsub",
        "target": "IotCore"
      }
    }
  }
  1. Granted an access policy to my custom component to allow publishing to IoT Core, strictly following the example in the link here
{ 
    "accessControl": {
      "CUSTOM_NAME": {
        "CUSTOM_NAME": {
          "policyDescription": "Allows access to publish to MY_TOPIC",
          "operations": [
            "aws.greengrass#PublishToTopic",
            "aws.greengrass#PublishToIoTCore"
          ],
          "resources": [
            "MY_TOPIC"
          ]
        }
      }
    }
}
answered 2 years ago
  • By the way, you don't need the merge in nucleus configuration if you've configured all of those details at the time of installing Greengrass.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions