Issue with adding a new greengrass core device to an existing deployment

0

Hi, I'm trying to write a script that is able to install and configure GreenGrass onto a blank debian install. My script successfully installs greengrass, names the device, and adds it to a deployment group. The issue I'm facing is after the deployment runs 'in progress' for about 5 minutes it will come back as failed, but if I rerun the deployment from the console to all the devices part of that deployment group without any changes to the configuration, it succeeds. What I notice that is odd is if I navigate to the artifacts directories for each component that is part of the deployment, they will all be empty but when the deployment succeeds, then they all populate.

The only oddity I can find in the logs is this in the greengrass.log

2023-09-01T17:24:15.244Z [INFO] (pool-2-thread-19) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=com.[REDACTED].[REDACTED], currentState=BROKEN}
2023-09-01T17:24:15.245Z [INFO] (pool-2-thread-19) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=com.[REDACTED].[REDACTED], currentState=BROKEN}
2023-09-01T17:24:16.058Z [WARN] (pool-2-thread-6) com.aws.greengrass.deployment.DeploymentConfigMerger: merge-config. merge-config-service BROKEN. {serviceName=com.[REDACTED].[REDACTED]}
2023-09-01T17:24:16.060Z [ERROR] (pool-2-thread-6) com.aws.greengrass.deployment.activator.DeploymentActivator: merge-config. Deployment failed. {deploymentId=6d8c996d-8e76-4a74-864d-0d93ea1cb8eb}
com.aws.greengrass.deployment.exceptions.ServiceUpdateException: Service com.[REDACTED].[REDACTED] in broken state after deployment
        at com.aws.greengrass.deployment.DeploymentConfigMerger.waitForServicesToStart(DeploymentConfigMerger.java:200)
        at com.aws.greengrass.deployment.activator.DefaultActivator.activate(DefaultActivator.java:83)
        at com.aws.greengrass.deployment.DeploymentConfigMerger.updateActionForDeployment(DeploymentConfigMerger.java:152)
        at com.aws.greengrass.deployment.DeploymentConfigMerger.lambda$mergeInNewConfig$0(DeploymentConfigMerger.java:104)
        at com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService.runUpdateActions(UpdateSystemPolicyService.java:95)
        at com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService.lambda$startup$0(UpdateSystemPolicyService.java:169)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Also if its of any help, this is the line from my install script that installs and configures greengrass:

java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region us-east-2 --thing-name [REDACTED] --thing-group-name TEST-MASS-DEPLOYMENT --component-default-user ggc_user:ggc_group --provision true --setup-system-service true --deploy-dev-tools true

Any help is greatly appreciated. Thank you!

asked a year ago382 views
2 Answers
1
Accepted Answer

As mentioned by @shagupta-aws the reason while your deployment fails is due to the component failing to correctly execute one of the lifecycle scripts. Check the logs generated by that component to understand the reason. The logs can be found in /greengrass/v2/logs.

The package folder is empty on the device where the deployment fails because you have selected Rollback as failure handling policy for the deployment. If you want Greengrass to retain the artifact of the component being deployed, select Don't rollback in the deployment policies section.

Enter image description here

AWS
EXPERT
answered a year ago
profile picture
EXPERT
reviewed 7 months ago
  • Ahh this was it, because it was rolling back, I didn't get a chance to see what was actually failing. Thank you!

1

Hi, the log you shared indicates that the component for which this log is, is failing to run. A lifecycle script for the component from its recipe is failing to execute, leading to it getting into broken state and the deployment failing. In order to understand why it fails, you should check the component's log file and observe any logs your component spits out or other information such as what script failed and with what exit code, etc. The other two things you mentioned will need more information in order for us to investigate and help you troubleshoot -

  1. why the deployment succeeds when you redeploy from the console without any change. Since we do not have the full greengrass log or component logs here on this post, we cannot identify the cause of these things. Is it possible for you to open a support case and attach all relevant logs and config files, etc to it?

  2. Artifact directory being empty at first and being populated later. It depends on when you observe it, initially on installation the directory will be empty and a deployment will download any artifacts for components as applicable so this should be as expected.

AWS
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions