Best way to troubleshoot Greengrass "queued" job? Device not receiving deployment.

0

I have a specific IOT greengrass device that I am able to access remotely through a 3rd party tunneling service. The latest deployment job is "queued" in the IOT dashboard, meaning that the device is waiting to receive the job documents. My current deployment revision is 50, but this specific device is deployed at 47. Dashboar status indicates HEALTHY but with a reported status of 3 months ago. I've double checked that the greengrass nucleus service is running, the certificates are loaded, and policies attached. The greengrass logs don't indicate any errors that I can troubleshoot. What is a good path to start down in order to determine why deployments are not being received on this device? Any help would be greatly appreciated.

1 Answer
1

Dashboar status indicates HEALTHY but with a reported status of 3 months ago

This should be updated daily, if you have default fleetStatus configuration in your nucleus: https://docs.aws.amazon.com/greengrass/v2/developerguide/greengrass-nucleus-component.html#greengrass-nucleus-component-configuration . This status is telling us your device hasn't communicated fleet status for 3 months, suggesting you have a connectivity problem that is about more than just deployments. I would expect you to have errors in greengrass.log, but it seems you don't.

In the IoT Core console, I suggest you go to Manage->All devices->Things-><thingName>->Activity tab. Using your SSH access, then restart the Greengrass service or reboot the device. If you have a healthy connection, you will see events in that Activity tab; unsubscribed, disconnected, connected and subscribed. Also the core device health should refresh. If you don't witness this, something fundamental is wrong. In this case, you should re-check the greengrass.log for errors or warnings, re-check IoT polices and certificates, check any firewall settings and so forth. Also ensure that the device your SSHing into is indeed the core device/Thing you're monitoring.

profile picture
EXPERT
Greg_B
answered 3 months ago
  • Thank you! Upon reboot/restarting Greengrass and inspecting the logs, I found some WARN level entries which indicate the issue here.

    com.aws.greengrass.deployment.IotJobsHelper: Device not configured to talk to AWS Iot cloud. IOT job deployment is offline: [IoT credential endpoint region [redacted].credentials.iot.us-west-2.amazonaws.com does not match the AWS region us-east-1 of the device]

    All of the greengrass config files indicate a us-west-2 region. Where would I update AWS region which is currently set to us-east-1? East-1 region is not being utilized in our account at all

  • Following up to let you know I have made progress on the issue. I am running Nucleus v2.0.5. No matter what I had in my config.yaml file, the config/effectiveConfig.yaml file always showed the incorrect region, which happened to be us-east-1 (the default region). I explored github and noticed there were some issues with v2.0.5 and region mapping, so I initiated the system service with the --aws-region flag.

    With that, the deployment succeed at is now up to date. However, the status in AWS IOT dashboard still remains as it was, with a status reported of 3 months ago.

  • The core device health is only updated every 24 hours (by default). So it may come good after that amount of time.

    I'm not sure what region problem you mean for 2.0.5. If possible, I would recommend you upgrade the Nucleus version.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions