AWS IoT Device Not Starting Remote Job

Based on the scenario you've described, there are several potential reasons why your device might be stuck in the "Queued" state for a remote job. Here are some possible causes and steps you can take to troubleshoot and remedy the situation:

1. Device not receiving job notifications: Even though the device is connected and sending other MQTT messages, it's possible that it's not receiving or processing the job notifications. Ensure that the device is properly subscribed to the correct job notification topics.

2. Permissions issue: Check that the IAM role associated with the job has the correct permissions to interact with the device and execute the job.

3. Device-side processing problem: There might be an issue in the device's code that's preventing it from acknowledging or starting the job execution. Review your device-side code to ensure it's correctly handling job notifications and updating job execution status.

4. Job document size: Verify that the job document size is within the MQTT payload size limits. If it's too large, the device might not be able to receive it.

5. Throttling or connection issues: While the device is connected, there might be intermittent connection issues or throttling occurring. Check your CloudWatch logs for any related errors.

To remedy the situation, try the following steps:

1. Review CloudWatch logs: Look for any error messages or unusual patterns related to the specific device and job execution.

2. Manually update job execution status: Use the AWS IoT Jobs API to manually update the job execution status from "Queued" to "In Progress" and see if this triggers any response from the device.

3. Check device logs: If possible, examine the logs on the device itself to see if there are any error messages or indications of why it's not processing the job.

4. Verify MQTT subscriptions: Ensure the device is correctly subscribed to all necessary MQTT topics for job notifications and updates.

5. Test with a new job: Create a new, simple job targeting only this device to see if it responds differently.

6. Device reboot: If possible, try rebooting the device to see if it resolves the issue.

7. Increase client-side timeout: If you're using a custom client, try increasing the client-side timeout values to account for potential latency issues.

If none of these steps resolve the issue, you may need to further investigate the device-side implementation to ensure it's correctly handling job notifications and updating job execution statuses.
Sources
AWS IoT Jobs Troubleshooting - AWS IoT Core
Jobs and job execution states - AWS IoT Core
Devices and jobs - AWS IoT Core

Hi. To progress from the job execution fromQUEUED state to IN_PROGRESS, the device needs to use the UpdateJobExecution operation described here: https://docs.aws.amazon.com/iot/latest/developerguide/jobs-mqtt-api.html

This means the device has to publish a message to the following topic: $aws/things/thingName/jobs/jobId/update. With the status key set to IN_PROGRESS.

If the job execution is staying stuck in the QUEUED state, possible causes would include:

1. The device doesn't receive the job on $aws/things/thingName/jobs/notify or $aws/things/thingName/jobs/notify-next topics.
2. It receives the job on those topics, but fails to publish a message on $aws/things/thingName/jobs/jobId/update to update the state of the job execution.
3. It does publish a message on that topic, but it's not correctly formed.

Do you have sufficient device logging to see what it's doing when it receives the job, and whether or what it's publishing?

You can also use CloudWatch to see what's happening with the job execution: https://docs.aws.amazon.com/iot/latest/developerguide/cwl-format.html#job-logs

In particular, this log entry to see if the device is updating the job execution state correctly or not: https://docs.aws.amazon.com/iot/latest/developerguide/cwl-format.html#log-job-update-job-ex.