IoT Custom job for running a ota update fails via aws-iot client service

Question

Hello, i have been using aws iot jobs for sending custom jobs like restarting a service, rebooting a device or running a custom ota job like pulling from git. I have noticed that when i run the built binary of aws-iot-client manually from a shell with the job's config file it works when it recieves a custom ota job. But when the same binary with same config file is run from aws iot service it doesn't work. The job reports failed state with a message code exited with status code 1. Can you help me more on this.

The update-software.sh file for ota handler is:

```
#!/usr/bin/env sh
set -e

echo "Running update-software.sh"
user=$1
echo "Username: $user"
if id "$user" 2>/dev/null && command -v "git" > /dev/null
then
    echo "Starting the update command"
    . /home/pi/triton_software/update.sh 
else
    "Update failed!!"
    exit 1
fi
echo "update-software command is done"

```

and the update.sh script which is being called is

```
#!/usr/bin/env sh
cd /home/pi/amazon-kinesis-video-streams-producer-sdk-cpp;
git pull origin main;
cd build/;
cmake -DBUILD_GSTREAMER_PLUGIN=TRUE ..;
make;

```

Any help is appreciated, Thanks
Subham

Accepted Answer

Tried it. I have changed a few things, but with success.

```
$ tree /work -L 2
```
```
/work
├── amazon-kinesis-video-streams-producer-sdk-cpp
│   ├── build
│   ├── certs
│   ├── CMake
│   ├── CMakeLists.txt
│   ├── dependency
│   ├── docs
│   ├── kvs_log_configuration
│   ├── LICENSE
│   ├── NOTICE
│   ├── open-source
│   ├── README.md
│   ├── samples
│   ├── src
│   └── tst
├── aws-iot-device-client
│   ├── build
│   ├── CHANGELOG.md
│   ├── CMakeLists.txt
│   ├── CMakeLists.txt.awssdk
│   ├── CMakeLists.txt.gtest
│   ├── CMakeLists.txt.versioning
│   ├── cmake-toolchain
│   ├── CODE_OF_CONDUCT.md
│   ├── config-template.json
│   ├── CONTRIBUTING.md
│   ├── docs
│   ├── format-check.sh
│   ├── LICENSE
│   ├── NOTICE
│   ├── old-schema-job-docs-and-handlers
│   ├── README.md
│   ├── run-cppcheck.sh
│   ├── sample-job-docs
│   ├── sample-job-handlers
│   ├── setup
│   ├── setup.sh
│   ├── source
│   └── test
├── cert
│   ├── AmazonRootCA1.pem
│   ├── certificate.pem.crt
│   ├── private.pem.key
│   └── public.pem.key
├── log
│   └── aws-iot-device-client.log
├── update.sh
└── update-software.sh

22 directories, 26 files
```

* /root/.aws-iot-device-client/aws-iot-device-client.conf

```json
    {
      "endpoint":       "abcdefg-ats.iot.us-east-1.amazonaws.com",
      "cert":   "/work/cert/certificate.pem.crt",
      "key":    "/work/cert/private.pem.key",
      "root-ca":        "/work/cert/AmazonRootCA1.pem",
      "thing-name":     "thing",
      "logging":        {
        "level":        "DEBUG",
        "type": "FILE",
        "file": "/var/log/aws-iot-device-client/aws-iot-device-client.log",
        "enable-sdk-logging":   false,
        "sdk-log-level":        "TRACE",
        "sdk-log-file": "/var/log/aws-iot-device-client/sdk.log"
      },
      "jobs":   {
        "enabled":      true,
        "handler-directory": "/root/.aws-iot-device-client/jobs"
      },
      "tunneling":      {
        "enabled":      false
      },
      "device-defender":        {
        "enabled":      false,
        "interval": 300
      },
      "fleet-provisioning":     {
        "enabled":      false,
        "template-name": "",
        "template-parameters": "",
        "csr-file": "",
        "device-key": ""
      },
      "samples": {
        "pub-sub": {
          "enabled": false,
          "publish-topic": "",
          "publish-file": "/root/.aws-iot-device-client/pubsub/publish-file.txt",
          "subscribe-topic": "",
          "subscribe-file": "/root/.aws-iot-device-client/pubsub/subscribe-file.txt"
        }
      },
      "config-shadow":  {
        "enabled":      false
      },
      "sample-shadow": {
        "enabled": false,
        "shadow-name": "",
        "shadow-input-file": "",
        "shadow-output-file": ""
      }
    }
```

* /work/update-software.sh

```sh
#!/usr/bin/env sh
set -e

echo "Running update-software.sh"
user=$1
echo "Username: $user"
if id "$user" 2>/dev/null && command -v "git" > /dev/null
then
    echo "Starting the update command"
    . /work/update.sh 
else
    echo "Update failed!!"
    exit 1
fi
```

* /work/update.sh

```sh
#!/usr/bin/env sh
cd /work/amazon-kinesis-video-streams-producer-sdk-cpp;
git pull origin master;
cd build/;
cmake -DBUILD_GSTREAMER_PLUGIN=TRUE ..;
make;
```

* Job document

```json
{
  "_comment": "sample",
  "version": "1.0",
  "steps": [
    {
      "action": {
        "name": "Git",
        "type": "runHandler",
        "input": {
          "handler": "update-software.sh",
          "path": "/work"
        },
        "runAsUser": "root"
      }
    }
  ]
}
```

* log

```
2022-04-30T13:22:27.922Z [DEBUG] {JobsFeature.cpp}: We have not seen a job yet, this is not a duplicate job notification
2022-04-30T13:22:27.922Z [INFO]  {JobsFeature.cpp}: Executing job: job2221
2022-04-30T13:22:27.922Z [DEBUG] {JobsFeature.cpp}: Attempting to update job execution status!
2022-04-30T13:22:27.922Z [DEBUG] {JobsFeature.cpp}: Not including stdout with the status details
2022-04-30T13:22:27.922Z [DEBUG] {JobsFeature.cpp}: Not including stderr with the status details
2022-04-30T13:22:27.922Z [DEBUG] {Retry.cpp}: Retryable function starting, it will retry until success
2022-04-30T13:22:27.923Z [INFO]  {JobEngine.cpp}: About to execute step with name: Git
2022-04-30T13:22:27.923Z [DEBUG] {JobEngine.cpp}: Using path {/work} supplied by job document for command execution
2022-04-30T13:22:27.923Z [INFO]  {JobEngine.cpp}: Did not find any arguments in the incoming job document. Value should be a JSON array of arguments
2022-04-30T13:22:27.923Z [INFO]  {JobEngine.cpp}: About to execute: /work/update-software.sh root 
2022-04-30T13:22:27.923Z [DEBUG] {JobsFeature.cpp}: Created EphemeralPromise for ClientToken yBZ1SqZjd0 in the updateJobExecution promises map
2022-04-30T13:22:27.923Z [DEBUG] {JobEngine.cpp}: Child process now running
2022-04-30T13:22:27.923Z [DEBUG] {JobEngine.cpp}: Child process about to call execvp
2022-04-30T13:22:27.923Z [DEBUG] {JobEngine.cpp}: Parent process now running, child PID is 21468
2022-04-30T13:22:27.924Z [DEBUG] {21468}: Running update-software.sh
2022-04-30T13:22:27.924Z [DEBUG] {21468}: Username: root
2022-04-30T13:22:27.927Z [DEBUG] {21468}: uid=0(root) gid=0(root) groups=0(root)
2022-04-30T13:22:27.927Z [DEBUG] {21468}: Starting the update command
...
2022-04-30T13:22:29.919Z [DEBUG] {21468}: update-software command is done
2022-04-30T13:22:29.919Z [DEBUG] {JobEngine.cpp}: JobEngine finished waiting for child process, returning 0
2022-04-30T13:22:29.919Z [WARN]  {JobEngine.cpp}: While executing action Git, JobEngine reported receiving errors from STDERR
2022-04-30T13:22:29.919Z [INFO]  {JobsFeature.cpp}: Job exited with status: 0
2022-04-30T13:22:29.919Z [WARN]  {JobsFeature.cpp}: JobEngine reported receiving errors from STDERR
2022-04-30T13:22:29.919Z [INFO]  {JobsFeature.cpp}: Job executed successfully!
2022-04-30T13:22:29.919Z [DEBUG] {JobsFeature.cpp}: Attempting to update job execution status!
2022-04-30T13:22:29.919Z [DEBUG] {Retry.cpp}: Retryable function starting, it will retry until success
2022-04-30T13:22:29.919Z [DEBUG] {JobsFeature.cpp}: Created EphemeralPromise for ClientToken QPIin7SC62 in the updateJobExecution promises map
2022-04-30T13:22:30.140Z [DEBUG] {JobsFeature.cpp}: Ack received for PublishUpdateJobExecutionStatus with code {0}
2022-04-30T13:22:30.140Z [DEBUG] {JobsFeature.cpp}: Removing ClientToken QPIin7SC62 from the updateJobExecution promises map
2022-04-30T13:22:30.140Z [DEBUG] {JobsFeature.cpp}: Success response after UpdateJobExecution for job job2221
2022-04-30T13:22:31.001Z [INFO]  {JobsFeature.cpp}: No pending jobs are scheduled, waiting for the next incoming job
```

Answer

Hi Subham. Just to clarify, you mean AWS IoT Device Client and you made your own job handler as outlined here? 
https://github.com/awslabs/aws-iot-device-client/blob/main/source/jobs/README.md#creating-your-own-job-handler

Do you get a **stderr** message as described here?
https://github.com/awslabs/aws-iot-device-client/blob/main/source/jobs/README.md#debugging-your-job

I think you need to narrow in on where it's failing. I'm unclear from your description of the error if **update-software.sh** is known to have successfully called **update.sh** or not. If it has, I think it should be possible to figure out, for example, whether the `git pull` worked or not. Also, you could add some more logging throughout **update.sh**.

IoT Custom job for running a ota update fails via aws-iot client service

Relevant content