Greengrass devices stuck on "Queued"

0

I have problems with deployment of greengrassV2 devices, here the procedure I follow: I install greengrass on a device from command line (linux) this sets up the device that is shown on aws web interface, so far so good. The device has now installed Nucleus and no error is showing on the logs. Mqtt connection is Ok. At this point as I have a deployment active for this device (for installing aws.greengrass.Cli) but it never executes. I have tried three times creating a virtual machine and installing greengrass but all three devices are stuck on "Queued" without any kind of clue about what is happening. Trying to resolve the issue I have revised and deleted deployments (is this a problem?). I have a deploying job on a "Thing group" and I have to say that deployment was working some time ago as I have installed components on devices, but now it does not work anymore. Logs on greengrass devices do not show errors or warnings, all devices status is reported on the web interface as healty and online. Thanks for any support.

Here the greengrass.log:

 [INFO] (main) com.aws.greengrass.util.platforms.Platform: Getting platform instance com.aws.greengrass.util.platforms.unix.linux.LinuxPlatform.. {}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: No ongoing deployment detected. Proceed as default. {}
 [INFO] (main) com.aws.greengrass.config.Configuration: config-loading. Read configuration from a file path. {path=/greengrass/v2/config/config.tlog}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/greengrass/v2/config/effectiveConfig.yaml}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: system-start. Launch Nucleus. {configPath=/greengrass/v2/config, rootPath=/greengrass/v2, version=2.12.1}
 [INFO] (main) software.amazon.awssdk.eventstreamrpc.RpcServer: IpcServer started.... {}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: serviceImplementors: {UpdateSystemPolicyService:class com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService...}. {}
 [INFO] (main) com.aws.greengrass.security.SecurityService: Register crypto key service provider. {keyType=file}
 [INFO] (main) com.aws.greengrass.security.SecurityService: Register MQTT connection security provider. {keyType=file}
 [INFO] (main) com.aws.greengrass.mqttclient.spool.Spool: Spooler has been configured. {maxSizeInBytes=2621440, storageType=Memory, keepQos0WhenOffline=false}
 [INFO] (main) com.aws.greengrass.mqttclient.spool.Spool: Memory Spooler has been set up. {}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=aws.greengrass.Nucleus}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=main}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=UpdateSystemPolicyService}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=DeploymentService}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=TelemetryAgent}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/greengrass/v2/config/effectiveConfig.yaml}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: system-start. {main=services.main:NEW}
 [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=NEW, newState=INSTALLED}
 [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=NEW, newState=INSTALLED}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=NEW, newState=INSTALLED}
 [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=NEW, newState=INSTALLED}
 [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=INSTALLED, newState=STARTING}
 [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=NEW, newState=INSTALLED}
 [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=INSTALLED, newState=STARTING}
 [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=STARTING, newState=RUNNING}
 [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: Stopping backingTask waiting for dependency ready. {serviceName=FleetStatusService, currentState=STARTING}
 [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=STARTING, newState=RUNNING}
 [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=INSTALLED, newState=STARTING}
 [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: Stopping backingTask waiting for dependency ready. {serviceName=DeploymentService, currentState=STARTING}
 [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=STARTING, newState=RUNNING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=INSTALLED, newState=STARTING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Stopping backingTask waiting for dependency ready. {serviceName=aws.greengrass.Nucleus, currentState=STARTING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=STARTING, newState=FINISHED}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=FINISHED, newState=STOPPING}
 [INFO] (pool-2-thread-11) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-finished. Nothing done. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
 [INFO] (pool-2-thread-8) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
 [INFO] (pool-2-thread-8) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Stopping backingTask start. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING, newState=FINISHED}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=NEW, newState=INSTALLED}
 [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=INSTALLED, newState=STARTING}
 [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: Stopping backingTask waiting for dependency ready. {serviceName=UpdateSystemPolicyService, currentState=STARTING}
 [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=STARTING, newState=RUNNING}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=INSTALLED, newState=STARTING}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Stopping backingTask waiting for dependency ready. {serviceName=main, currentState=STARTING}
 [INFO] (pool-2-thread-11) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-finished. Nothing done. {serviceName=main, currentState=STARTING}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=STARTING, newState=FINISHED}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=FINISHED, newState=STOPPING}
 [INFO] (pool-2-thread-11) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=main, currentState=STOPPING}
 [INFO] (pool-2-thread-11) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=main, currentState=STOPPING}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=STOPPING, newState=FINISHED}
 [INFO] (main) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {trigger=NUCLEUS_LAUNCH, serviceName=FleetStatusService, currentState=RUNNING}
 [INFO] (Thread-4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Successfully connected to AWS IoT Core. {clientId=X-Gate-tmp-test-1, sessionPresent=false}

Greengrass core device status

Thing group

Greengrass Deployment

asked a month ago309 views
5 Answers
0
Accepted Answer

Hi

Firstly, I want to thank you for your patience and cooperation in this issue. This is certainly a weird one. From the debug logs, I can see that your device is successfully subscribing to deployment job notifications, however, the device never receives those notifications. Greengrass will not do anything proactively if does not receive the notifications to begin with and that is why your deployment remains in QUEUED state forever.

As a last ditch effort, please confirm that GreengrassV2IoTThingPolicy on your IoT console exists and is configured correctly. You can find this policy under the Policies section in the Security drop down in the navigation panel on the left side of your AWS console. The policy should look something like

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Connect",
        "iot:Publish",
        "iot:Subscribe",
        "iot:Receive",
        "greengrass:*"
      ],
      "Resource": "*"
    }
  ]
}

If the policy looks good, you should open a ticket to CTI (Category, Type, Item) AWS, IoT, Jobs for further assistance.

Siddhant

AWS
sidsriv
answered a month ago
  • Seems like you found the issue... Checking my policy, for some reason iot:Receive was missing. Fixing this solved the issue. Many thanks for YOUR patience and compentence

0

Hi

Thanks for reaching out to us. From the looks of things, your Greengrass device is unable to receive the deployments you created on the Greengrass console. Could you please

  1. check if your deployment targets the correct thing/thing group?
  2. check if your device has a stable network connection?

Thanks, Siddhant

AWS
sidsriv
answered a month ago
  • I think that both of the checks are ok. I edited the answer with more info about the issue. Really thanks for your help!

0

Hello,

A deployment stuck on "Queued" means the device has not received the deployment. Based on your scenario, you confirmed the connection is OK by looking at MQTT connection, and could you confirm the following please?

  1. When the device status is last updated?
  2. Is the thing a member of your targeted "Thing group"?

For further analysis, could you share us with your greengrass.log please? Please do not include any sensitive info.

Thanks

AWS
yitingb
answered a month ago
  • Hi, I updated the question attaching greengrass device log and some snippets that shows:

    • Device last update and
    • IoT Thing group
    • Info about the deployment Really thanks for your help!
0

Thanks for bringing to our attention, this certainly does not look right. Could you please confirm if the logs provided in the snippet provided are complete? If they are, your device does not seem to be subscribing to IoT Jobs as expected and that is causing it to not receive deployments. Could you please try out a couple of things for me:

  1. Please look up 'IotJobsHelper' in your logs from all the 4 devices in the target thing group. Please share if you find anything.
  2. Please provide the exact command used to install Greengrass on any of your device.
  3. Please try to install Greengrass on a fresh device the same way you have been installing so far except this time, enable debug logs by using -Dlog.level=DEBUG in the installation command. Please provide the logs for this simulation as well.

Thanks, Siddhant

AWS
sidsriv
answered a month ago
  • Thanks for the reply, I tried to answer to all questions

0

Hi Siddhant,

Could you please confirm if the logs provided in the snippet provided are complete?

Yes

  1. Please look up 'IotJobsHelper' in your logs from all the 4 devices in the target thing group. Please share if you find anything.
  • On one device that previously had successful deployment and now is not anymore updating, I found that doing cat /greengrass/v2/logs/* | grep IotJobs gives me results for days ago, not giving results for recent logs.
  • On another greengrass device I created from scratch recently where I had no succesful deployment I cannot find IotJobs.
  • In the device that I created from scratch as you requested (point 3 of your questions) I find this:
root@host:~# cat /greengrass/v2/logs/* | grep IotJobs
2024-01-31T10:06:06.129Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribing to Iot Jobs Topics. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.129Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribing to deployment job execution update.. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.537Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribed to deployment job execution update.. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.537Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribing to deployment job event notifications.. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.596Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribed to deployment job event notifications.. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.607Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Requesting the next deployment. {ThingName=GreengrassQuickStartCore-18d5ef74879}
  1. Please provide the exact command used to install Greengrass on any of your device.

Normally I take it from management console, like this:

sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region eu-central-1 --thing-name MyThingName --thing-group-name X-Gate --component-default-user ggc_user:ggc_group --provision true --setup-system-service true
  1. Please try to install Greengrass on a fresh device the same way you have been installing so far except this time, enable debug logs by using -Dlog.level=DEBUG in the installation command. Please provide the logs for this simulation as well.
root@host:~# apt install -y openjdk-17-jre-headless

root@host:~# sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -Dlog.level=DEBUG -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region eu-central-1 --thing-name GreengrassQuickStartCore-18d5ef74879 --thing-group-name X-Gate --component-default-user ggc_user:ggc_group --provision true --setup-system-service true
Provisioning AWS IoT resources for the device with IoT Thing Name: [GreengrassQuickStartCore-18d5ef74879]...
Found IoT policy "GreengrassV2IoTThingPolicy", reusing it
Creating keys and certificate...
Attaching policy to certificate...
Creating IoT Thing "GreengrassQuickStartCore-18d5ef74879"...
Attaching certificate to IoT thing...
Successfully provisioned AWS IoT resources for the device with IoT Thing Name: [GreengrassQuickStartCore-18d5ef74879]!
Adding IoT Thing [GreengrassQuickStartCore-18d5ef74879] into Thing Group: [X-Gate]...
IoT Thing Group "X-Gate" already existed, reusing it
Successfully added Thing into Thing Group: [X-Gate]
Setting up resources for aws.greengrass.TokenExchangeService ... 
Attaching TES role policy to IoT thing...
No managed IAM policy found, looking for user defined policy...
IAM policy named "GreengrassV2TokenExchangeRoleAccess" already exists. Please attach it to the IAM role if not already
Configuring Nucleus with provisioned resource details...
Downloading Root CA from "https://www.amazontrust.com/repository/AmazonRootCA1.pem"
Created device configuration
Successfully configured Nucleus with provisioned resource details!
Successfully set up Nucleus as a system service

Log of the device Log of the device

I have noticed some errors on Mqtt regarding the shadow of the device, is this normal? Or how can be fixed?

$aws/things/X-Gate-18b1dfd630b-gci/shadow/get/rejected
January 31, 2024, 11:50:19 (UTC+0100)
{
  "code": 404,
  "message": "No shadow exists with name: 'X-Gate-18b1dfd630b-gci'"
}
$aws/things/X-Gate-18b1dfd630b-gci/shadow/get
January 31, 2024, 11:51:13 (UTC+0100)
{
  "thingName": "X-Gate-18b1dfd630b-gci"
}

In the meanwhile the deployment execution is stil Queued for all devices.

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions