IDT 4.5.11 Error in retrieving AwsCredentials from TES

Question

I am using IDT 4.5.11 running test suite GGV2Q_2.4.1 on Greengrass 2.7.0. 
I failed the [mqttpubsub] and [cloudcomponent] test groups.
Below are logs from mqttpubsub/greengrass.log, I am not sure what credential is expired as the Access & Secret Key are both newly created.
Any idea how to solve this issue?

2022-11-21T19:04:13.050Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.tes.CredentialRequestHandler: Received response from cloud: response code 200, not logging credentials. {iotCredentialsPath=/role-aliases/idt-a1a2f5615374392fcbc9-ggc-role-alias/credentials, statusCode=200}
2022-11-21T19:04:13.050Z [ERROR] (pool-2-thread-6) com.aws.greengrass.tes.CredentialRequestHandler: Unable to cache expired credentials which expired at 2022-11-21T12:03:44Z. {iotCredentialsPath=/role-aliases/idt-a1a2f5615374392fcbc9-ggc-role-alias/credentials}
2022-11-21T19:04:13.051Z [ERROR] (pool-2-thread-6) com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES. {iotCredentialsPath=/role-aliases/idt-a1a2f5615374392fcbc9-ggc-role-alias/credentials, credentialData=TES responded with expired credentials: {"credentials":{"accessKeyId":"ASIAZ625MVFNBTBLFT6Z","secretAccessKey":"WD5VbRjGmp3mDhqSL6udPjID9cZ+b2WEQ4Rf/CWu","sessionToken":"IQoJb3JpZ2luX2VjEIP//////////wEaCXVzLXdlc3QtMiJIMEYCIQCJXJM5KGN71gUQvo5F5j7T2n74d4qc+i066ZDrDIYOpgIhAPJG1U4LyhLFkOjQQbCo9Bq1ht9boqtfiQUE2I03P/OaKqwECIz//////////wEQABoMNjg0NzA2Mjc3NzIyIgzH/z2OTB04Urnlr3wqgARdz+wbs2oS/Hr1h2Ep8KCHqbBaixaDRDCa3HawQkI+UW3r+A7JcXiiM2MZcNNjgr2X+YOdgwpPA6Pd+3b7V8fXcQudkxQfSCXJcoB0gssCi/IrlNe8mdRViN/hfxNIk1H91RxygSE/epH+f2WEfh7lt81ZUNTelBfxor36InjawZu3t0ZLv74ZIlJBcjxNSajKaxg8lKB+sxoQzaF6OlR0nrkx2W0Np9ikFk1eYeWCWyOgnQ8PhcRKivTU77b7KXlOCCkkpITxMU0/gf8EuWfsy9Ik9KQZjr/CNxgvIo8nE7VE0Be1TEvRdRwP5VsH3+EAxCv5VJkXFg8vka7R01JKkweDXEb3JVJhcKz82xr9hIFo5an34InDbc3ssC/I19/ZSZHygVE+aEjgujpmrhLkDUThSsjm8E6+rHdDoY3W+ugL2ik/Yjwk04jT/nl49T86woRk+bU08VW03Vwk+kfqpMX34mP4FntuChc8Niig3E0jWznlX4N595tg+LUj6LXjpaYFNkQbena1b7iXalA+zxHw11GS+V5oisipH9zsCNrdI9hhzqpPt+YhyPCLg28H7hXNLUezxdJAaVjlAuOS8SUR9R35A+POFuUp5T0nLi5XutGPq7r2q+kh1RXCbLKlCM4ajyejb6jsr4tlyo/VUPY/FVXghoIVaPDOBUHh3zCQtu2bBjrFAcsqM7H6gspDrVero0uPfAuHgExojSa3UsIu6kgEWmPJeFTonceeZemuQp8flqxLsdnRcC5IAHd19XA3w35MWGZ+9tcSOVYSpzYepdXw31iVMoFqs1bYLF5NtbqlTZKHrM1O7M5rZsyH7Z2D4JzMMqAwGUhb4bdpp+1Aouvlpm6NE63FPqrvsh0SNpGurRYE2Ijwtj0Kmt1Q+oM0uxrh+gZB77aWGxzjVnQORpoVGWn6oynNxR/LlI5LfmGfDawKMozMQxpk","expiration":"2022-11-21T12:03:44Z"}}}
2022-11-21T19:04:13.051Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.componentmanager.builtins.S3Downloader: get-bucket-location. task failed and will be retried. {task-attempt=7, componentIdentifier=aws.greengrass.IotMqttSubscriber, artifactUri=s3://idt-a1a2f5615374392fcbc9-gg-component-store/greengrass/components/artifacts/aws-greengrass-testing-features-mqtt.zip}
software.amazon.awssdk.core.exception.SdkClientException: Failed to fetch credentials
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:39)
        at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:31)
        at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.resolveCredentials(AwsExecutionContextBuilder.java:171)
        at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:108)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
        at software.amazon.awssdk.services.s3.DefaultS3Client.getBucketLocation(DefaultS3Client.java:3382)
        at com.aws.greengrass.componentmanager.builtins.S3Downloader.lambda$getRegionClientForBucket$2(S3Downloader.java:134)
        at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:50)
        at com.aws.greengrass.componentmanager.builtins.S3Downloader.getRegionClientForBucket(S3Downloader.java:133)
        at com.aws.greengrass.componentmanager.builtins.S3Downloader.getDownloadSize(S3Downloader.java:115)
        at com.aws.greengrass.componentmanager.ComponentManager.prepareArtifacts(ComponentManager.java:420)
        at com.aws.greengrass.componentmanager.ComponentManager.preparePackage(ComponentManager.java:377)
        at com.aws.greengrass.componentmanager.ComponentManager.lambda$preparePackages$1(ComponentManager.java:338)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Answer

Hello,

I could see that, 'mqttpubsub' and 'cloudcomponent' test groups are being failed when running running test suite GGV2Q_2.4.1 on Greengrass 2.7.0 using IDT 4.5.1. On checking the logs, you noticed that the TES component responded with expired AWS credentials.

To begin with can you please confirm if the clock on the device is correct. Also, to further investigate the issue, I would need to take a look at the IDT[1] and Greengrass[2] logs generated pertaining to this specific issue. You can create a technical case using AWS Support console and share us all the above mentioned logs.

References:

[1]. https://docs.aws.amazon.com/greengrass/v2/developerguide/idt-troubleshooting.html#where-to-look

[2]. https://docs.aws.amazon.com/greengrass/v2/developerguide/monitor-logs.html#access-local-logs

Answer

Maybe the credentials are expiring before the tests are able to complete? You can try the following things:

* Request a longer expiration period for the temporary credentials. The default is 15 minutes. The max can be up to 12 hours usually. See https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken for more info.
* Run only one of the failing test cases, as it should run more quickly than running multiple test cases back to back (if that's what you were doing before), to see if the test case is only failing because of the timeout. You can do this by providing a group-id and test-id (test case id) arguments to the run-suite command. See https://docs.aws.amazon.com/greengrass/v2/developerguide/run-tests.html for more info.

IDT 4.5.11 Error in retrieving AwsCredentials from TES

Relevant content