- Newest
- Most votes
- Most comments
Hello @loordb, Looking at your provided information I would suggest restarting greengrass and then re-running the tests again with the timeout-multiplier flag. This will increase the timeout of the tests and make sure what you are running into is not just simply a time issue.
To restart greengrass:
sudo service greengrass restart
To re-run tests with the timeout-multiplier:
./devicetester_linux_x86-64 run-suite --suite-id GGV2Q_2.5.0 --timeout-multiplier 5 --userdata userdata.json
If you are still running into an error, can you please post your latest greengrass_YYYY_MM_DD_HH_X.log again along side the deploymentId and the deployment lifecycle phases the device has gone through, and the timestamps for the phase transition.
Knowing this information will tell us if you are actually running into an error or if it is just a time issue.
Regards.
I think the security credential cause the problem see the answer by the following link
Hello @jrcarb-AWS after trying the steps you have mentioned
I have upload the full log in
https://github.com/yuchinchenTW/IDT_-forgreengrassv2_log/tree/123/20230330T093130
hope it helps to clarify the error
sudo service greengrass restart
./devicetester_linux_x86-64 run-suite --suite-id GGV2Q_2.5.0 --timeout-multiplier 5 --userdata userdata.json
the test_manager.log shows the same error:
10:31:25.930 [otf-1.0.0-SNAPSHOT] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed GreengrassDeployment in GreengrassV2Lifecycle
10:31:26.167 [otf-1.0.0-SNAPSHOT] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed GreengrassDeployment in GreengrassV2Lifecycle
10:31:26.493 [otf-1.0.0-SNAPSHOT] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed GreengrassComponent in GreengrassV2Lifecycle
10:31:26.774 [otf-1.0.0-SNAPSHOT] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed GreengrassComponent in GreengrassV2Lifecycle
10:31:27.569 [otf-1.0.0-SNAPSHOT] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed S3Object in S3Lifecycle
10:31:28.173 [otf-1.0.0-SNAPSHOT] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed S3Bucket in S3Lifecycle
10:31:28.179 [otf-1.0.0-SNAPSHOT] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.features.AWSResourcesSteps - Successfully removed externally created resources
10:31:28.182 [otf-1.0.0-SNAPSHOT] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.features.LoggerSteps - Clearing thread context on scenario: 'Component publishes MQTT message to Iot core and retrieves it as well'
10:31:28.186 [] [mqtt] [idt-e44384162d7530cffc52] [INFO] com.aws.greengrass.testing.launcher.reporting.StepTrackingReporting - Passed: 'Component publishes MQTT message to Iot core and retrieves it as well'
[INFO] [2023-03-30 10:31:32]: All tests finished. executionId=98b8bcae-ce9a-11ed-8691-080027641c32
[INFO] [2023-03-30 10:31:34]:
========== Test Summary ==========
Execution Time: 59m58s
Tests Completed: 7
Tests Passed: 6
Tests Failed: 1
Tests Skipped: 0
----------------------------------
Test Groups:
lambdadeployment: PASSED
mqtt: PASSED
pretestvalidation: PASSED
coredependencies: PASSED
version: PASSED
component: FAILED
----------------------------------
Failed Tests:
Group Name: component
Test Name: cloudcomponent
Reason: Failed at 'the Greengrass deployment is COMPLETED on the device after 180 seconds'
----------------------------------
Path to AWS IoT Device Tester Report: /home/user/test/devicetester_greengrass_v2_linux/results/20230330T093130/awsiotdevicetester_report.xml
Path to Test Execution Logs: /home/user/test/devicetester_greengrass_v2_linux/results/20230330T093130/logs
Path to Aggregated JUnit Report: /home/user/test/devicetester_greengrass_v2_linux/results/20230330T093130/GGV2Q_Report.xml
==================================
here is the log you asked not sure if it help because after hours of search seems nothing infom for me greengrass_2023_03_30_01_0.log see the log in the link due to the log is too long and cannot post in the comment
it seems like the error shows the same at greengrass-test-run.log
2023-Mar-30 10:01:55,825 [cloudComponent] [idt-63ea438cc5fbbf1a023d] [INFO] com.aws.greengrass.testing.features.DeploymentSteps - Created Greengrass deployment: cd08a43f-4ffe-4849-aebb-de46d9fc5ee9
2023-Mar-30 10:01:55,826 [cloudComponent] [idt-63ea438cc5fbbf1a023d] [INFO] greengrass/features/cloudComponent.feature - Finished step: 'I deploy the Greengrass deployment configuration to thing group' with status PASSED
2023-Mar-30 10:16:56,096 [cloudComponent] [idt-63ea438cc5fbbf1a023d] [ERROR] greengrass/features/cloudComponent.feature - Failed at step: 'the Greengrass deployment is COMPLETED on the device after 180 seconds'
java.lang.IllegalStateException: Deployment idt-63ea438cc5fbbf1a023d-gg-deployment did not reach COMPLETED
at com.aws.greengrass.testing.features.DeploymentSteps.deploymentSucceeds(DeploymentSteps.java:311) ~[AWSGreengrassV2TestingIDT-1.0.jar:?]
at ✽.the Greengrass deployment is COMPLETED on the device after 180 seconds(classpath:greengrass/features/cloudComponent.feature:26) ~[?:?]
notice that the idt environment and greengrass device do have the time difference (UTC+8 && UTC+0 ;8 hours diff) but we think that the time differ does not causing the result of the error I mean it should work even if the device is in different places
Hello @loordb,
First off, thank you for submitting all of your logs, it was really helpful in debugging the issue. From your cloudcomponent/greengrass.log logs we can see that the component deployment started at around 1:58, and finished deployment successfully at around 2:31. Which means the deployment took around more than 30 mins, so my previous timeout multiplier was not enough. Can you please try again with a timeout multiplier of 12?
./devicetester_linux_x86-64 run-suite --suite-id GGV2Q_2.5.0 --timeout-multiplier 12 --userdata userdata.json
Regards.
Hello @jrcarb-AWS thx for the early reply, we have tried the command you have mentioned see the result below, seems like the same error still happening, we also have upload the new tesing log to git hope it helps
Hello @jrcarb-AWS after trying with the command which you have mentioned
./devicetester_linux_x86-64 run-suite --suite-id GGV2Q_2.5.0 --timeout-multiplier 12 --userdata userdata.json
seems like the same error still happened
after couple times of trying
the log have upload to the link below
https://github.com/yuchinchenTW/IDT_-forgreengrassv2_log/tree/123/20230331T102452
12:26:58.321 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.DefaultGreengrass - Leaving Greengrass running on pid: 0
12:26:58.322 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.features.FileSteps - Stopping Greengrass service..
12:27:24.342 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.features.FileSteps - Starting Greengrass service..
12:27:24.896 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed GreengrassDeployment in GreengrassV2Lifecycle
12:27:25.215 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed GreengrassDeployment in GreengrassV2Lifecycle
12:27:25.437 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed GreengrassDeployment in GreengrassV2Lifecycle
12:27:25.711 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed GreengrassComponent in GreengrassV2Lifecycle
12:27:26.373 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed S3Object in S3Lifecycle
12:27:26.883 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Removed S3Bucket in S3Lifecycle
12:27:26.884 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.features.AWSResourcesSteps - Successfully removed externally created resources
12:27:26.885 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-07ade2fc27374da68d46] [INFO] com.aws.greengrass.testing.features.LoggerSteps - Clearing thread context on scenario: 'As a developer, I can create a component in Cloud and deploy it on my device via thing group'
12:27:26.885 [] [cloudComponent] [idt-2e3f3b9d5e6bd3bcce32] [INFO] com.aws.greengrass.testing.launcher.reporting.StepTrackingReporting - Passed: 'As a developer, I can create a component in Cloud and deploy it on my device'
12:27:26.886 [] [cloudComponent] [idt-07ade2fc27374da68d46] [ERROR] com.aws.greengrass.testing.launcher.reporting.StepTrackingReporting - Failed: 'As a developer, I can create a component in Cloud and deploy it on my device via thing group': Failed at 'the Greengrass deployment is COMPLETED on the device after 180 seconds'
12:27:27.131 [] [] [] [INFO] com.aws.greengrass.testing.modules.AWSResourcesCleanupModule - Cleaned up TestContext{testId=TestId{prefix=idt, id=2e3f3b9d5e6bd3bcce32}, testDirectory=/tmp/gg-testing-6897313184919651577/idt-2e3f3b9d5e6bd3bcce32, testResultsPath=/home/user/test/devicetester_greengrass_v2_linux/results/20230331T102452/logs/component/cloudcomponent, cleanupContext=CleanupContext{persistAWSResources=false, persistInstalledSoftware=true, persistGeneratedFiles=false}, initializationContext=InitializationContext{persistModes=[installed.software], persistAWSResources=false, persistInstalledSoftware=true, persistGeneratedFiles=false}, logLevel=DEBUG, installRoot=/test/greengrass/v2, currentUser=ggc_user, coreThingName=IM30, coreVersion=2.9.4, tesRoleName=GreengrassV2TokenExchangeRole, hsmConfigured=false, trustedPluginsPaths=[]}
[ERROR] [2023-03-31 12:27:30]: Test exited unsuccessfully executionId=38162d05-cf6b-11ed-b7ea-080027641c32 testCaseId=cloudcomponent error=exit status 1
[INFO] [2023-03-31 12:27:30]: All tests finished. executionId=38162d05-cf6b-11ed-b7ea-080027641c32
[INFO] [2023-03-31 12:27:33]:
========== Test Summary ==========
Execution Time: 2h2m32s
Tests Completed: 7
Tests Passed: 6
Tests Failed: 1
Tests Skipped: 0
----------------------------------
Test Groups:
pretestvalidation: PASSED
coredependencies: PASSED
version: PASSED
mqtt: PASSED
lambdadeployment: PASSED
component: FAILED
----------------------------------
Failed Tests:
Group Name: component
Test Name: cloudcomponent
Reason: Failed at 'the Greengrass deployment is COMPLETED on the device after 180 seconds'
----------------------------------
Path to AWS IoT Device Tester Report: /home/user/test/devicetester_greengrass_v2_linux/results/20230331T102452/awsiotdevicetester_report.xml
Path to Test Execution Logs: /home/user/test/devicetester_greengrass_v2_linux/results/20230331T102452/logs
Path to Aggregated JUnit Report: /home/user/test/devicetester_greengrass_v2_linux/results/20230331T102452/GGV2Q_Report.xml
==================================
also upload our config help here hope it helps
config.json
{
"log": {
"location": "../logs/"
},
"configFiles": {
"root": "../configs",
"device": "../configs/device.json"
},
"testPath": "../tests/",
"reportPath": "../results/",
"certificatePath": "../certificates/",
"awsRegion": "us-west-2",
"auth": {
"method": "environment"
}
}
device.json
[
{
"id": "IM30",
"sku": "IM30",
"features": [
{
"name": "arch",
"value": "aarch64"
},
{
"name": "ml",
"value": "no"
},
{
"name": "docker",
"value": "no"
},
{
"name": "streamManagement",
"value": "no"
},
{
"name": "hsi",
"value": "no"
}
],
"devices": [
{
"id": "test",
"operatingSystem": "Linux",
"connectivity": {
"protocol": "ssh",
"ip": "192.168.120.29",
"port": 22,
"auth": {
"method": "password",
"credentials": {
"user": "root",
"password": "123"
}
}
}
}
]
}
]
later on we also have tried ./devicetester_linux_x86-64 run-suite --suite-id GGV2Q_2.5.0 --timeout-multiplier 20 --userdata userdata.json and the log shows the same error . Notice that we disconnect the ssh while after we see the same cloudcomponent error log in the log (because it take too long to test in --timeout-multiplier 20) so it will show the ssh disconnect log at the end
17:45:40.765 [otf-1.0.0-SNAPSHOT] [cloudComponent] [idt-4c42025f3cf8ba495b5d] [ERROR] com.aws.greengrass.testing.idt.IDTDevice - Failed to execute a command on null, CommandInput{line=sh, workingDirectory=null, input=null, timeout=null, args=[-c, systemctl stop greengrass.service]} com.amazonaws.iot.idt.exception.IDTServiceException: failed to execute on device: couldn't connect a remote session with error: ssh: unexpected packet in response to channel open: <nil> at com.amazonaws.iot.idt.IDTCredentialsInterceptor.enrich(IDTCredentialsInterceptor.java:73) at com.amazonaws.iot.idt.IDTCredentialsInterceptor.intercept(IDTCredentialsInterceptor.java:56)Before that all of the log seeems the same see the --timeout-multiplier 20 log in the link https://github.com/yuchinchenTW/IDT_-forgreengrassv2_log/tree/123/20230331T141120/logs
Hello
Based on the logs, after the timeout multiplier of 20 is applied, the cloud component test indeed went through all the steps without any issue. But, something triggered the retry of the same test, and during the retry the ssh timeout led to the failure.
My hypothesis is that, since this is a preinstalled Greengrass running for long time, the accumulated logs are coming in the way of the testcases to run properly. While IDT/Greengrass teams look into hardening the testcases in this particular usecase, I have one suggestion for you to try.
Can you please stop Greengrass, remove all the logs in logs folder, and restart the Greengrass installation, before running the tests again, with 20 multiplier.
Relevant content
- Accepted Answerasked a year ago
AWS OFFICIALUpdated 10 months ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 4 years ago
HI @jrcarb-AWS I have upload the log below after following the steps you mentions help it can clarify the problem thx