Creating python greengrass scripts that depend on other packages

Question

GG 2.7
Python 3.9.2
Raspberry Pi

I am learning GreenGrass. I tried [this tutorial](https://docs.aws.amazon.com/greengrass/v2/developerguide/ipc-iot-core-mqtt.html) and ran into an issue. My GG core device cannot find the awsiot imports at the top of the script. I am new to python (lot's of C# experience).  I think Visual Studio did this pretty much automatically. I tried installing awsiot on the pi directly but that did not solve the issue. I am assuming I have to package my script with dependencies somehow. How?

One note - I assume this will entail uploading artifacts to S3 somehow. I'd rather let PyCharm take care of creating a package for me rather than have to do a lot of hand editing of configuration files etc. Is there a way to "package" a python script to upload as a single file?

The code causing the errors:

```
import awsiot.greengrasscoreipc
import awsiot.greengrasscoreipc.client as client
from awsiot.greengrasscoreipc.model import (
    QOS,
    PublishToIoTCoreRequest
)
```
An example error:

> 2022-08-22T17:10:50.571Z [WARN] (Copier) com.example.publish: stderr. import awsiot.greengrasscoreipc. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}

**EDIT**
Adding to my question as there is too much detail to add a comment:
Not having a great experience with greengrass so far. I updated my publish.py script to be a one liner:

```
import aws.greengrass.SNS
```
My recipe looks like this:

```
{
  "RecipeFormatVersion": "2020-01-25",
  "ComponentName": "com.example.publish",
  "ComponentVersion": "1.0.5",
  "ComponentType": "aws.greengrass.generic",
  "ComponentDescription": "My first Greengrass component.",
  "ComponentPublisher": "Me",
  "ComponentConfiguration": {
    "DefaultConfiguration": {}
  },
  "ComponentDependencies": {
    "aws.greengrass.SNS": {
      "VersionRequirement": ">0.0.0",
      "DependencyType": "HARD"
    }
  },
  "Manifests": [
    {
      "Platform": {
        "os": "linux"
      },
      "Name": "Linux",
      "Lifecycle": {
        "Run": "python3  {artifacts:path}/publish.py '{configuration:/Message}'"
      },
      "Artifacts": [
        {
          "Uri": "s3://greengrass-component-artifacts-999999999999999-us-east-1/artifacts/com.example.publish/1.0.5/publish.py",
          "Digest": "pOKWcYoGpF1Ziy3K5nn0fzIE8xTeZJQ2BHn5tTe3Dgw=",
          "Algorithm": "SHA-256",
          "Unarchive": "NONE",
          "Permission": {
            "Read": "OWNER",
            "Execute": "NONE"
          }
        }
      ]
    }
  ],
  "Lifecycle": {}
}
```
And I end up with the following (that look like errors to me):

```
2022-08-26T13:36:44.390Z [INFO] (pool-2-thread-122) com.example.publish: shell-runner-start. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=STARTING, command=["python3  /greengrass/v2/packages/artifacts/com.example.publish/1.0.5/publish.p..."]}
2022-08-26T13:36:44.495Z [WARN] (Copier) com.example.publish: stderr. Traceback (most recent call last):. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.497Z [WARN] (Copier) com.example.publish: stderr. File "/greengrass/v2/packages/artifacts/com.example.publish/1.0.5/publish.py", line 1, in . {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.498Z [WARN] (Copier) com.example.publish: stderr. import aws.greengrass.SNS. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.499Z [WARN] (Copier) com.example.publish: stderr. ModuleNotFoundError: No module named 'aws'. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.514Z [INFO] (Copier) com.example.publish: Run script exited. {exitCode=1, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.521Z [INFO] (pool-2-thread-120) com.example.publish: shell-runner-start. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=STARTING, command=["python3  /greengrass/v2/packages/artifacts/com.example.publish/1.0.5/publish.p..."]}
2022-08-26T13:36:44.621Z [WARN] (Copier) com.example.publish: stderr. Traceback (most recent call last):. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.622Z [WARN] (Copier) com.example.publish: stderr. File "/greengrass/v2/packages/artifacts/com.example.publish/1.0.5/publish.py", line 1, in . {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.623Z [WARN] (Copier) com.example.publish: stderr. import aws.greengrass.SNS. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.624Z [WARN] (Copier) com.example.publish: stderr. ModuleNotFoundError: No module named 'aws'. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.641Z [INFO] (Copier) com.example.publish: Run script exited. {exitCode=1, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.649Z [INFO] (pool-2-thread-120) com.example.publish: shell-runner-start. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=STARTING, command=["python3  /greengrass/v2/packages/artifacts/com.example.publish/1.0.5/publish.p..."]}
2022-08-26T13:36:44.749Z [WARN] (Copier) com.example.publish: stderr. Traceback (most recent call last):. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.750Z [WARN] (Copier) com.example.publish: stderr. File "/greengrass/v2/packages/artifacts/com.example.publish/1.0.5/publish.py", line 1, in . {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.751Z [WARN] (Copier) com.example.publish: stderr. import aws.greengrass.SNS. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.752Z [WARN] (Copier) com.example.publish: stderr. ModuleNotFoundError: No module named 'aws'. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:44.769Z [INFO] (Copier) com.example.publish: Run script exited. {exitCode=1, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:45.223Z [INFO] (pool-2-thread-120) com.example.publish: shell-runner-start. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=STARTING, command=["python3 /greengrass/v2/packages/artifacts/com.example.publish/1.0.0/publish.py..."]}
2022-08-26T13:36:45.384Z [WARN] (Copier) com.example.publish: stderr. python3: can't open file '/greengrass/v2/packages/artifacts/com.example.publish/1.0.0/publish.py': [Errno 2] No such file or directory. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:45.400Z [INFO] (Copier) com.example.publish: Run script exited. {exitCode=2, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:45.753Z [INFO] (pool-2-thread-120) com.example.publish: shell-runner-start. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=STARTING, command=["python3 /greengrass/v2/packages/artifacts/com.example.publish/1.0.0/publish.py..."]}
2022-08-26T13:36:45.852Z [WARN] (Copier) com.example.publish: stderr. python3: can't open file '/greengrass/v2/packages/artifacts/com.example.publish/1.0.0/publish.py': [Errno 2] No such file or directory. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:45.869Z [INFO] (Copier) com.example.publish: Run script exited. {exitCode=2, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:46.151Z [INFO] (pool-2-thread-120) com.example.publish: shell-runner-start. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=STARTING, command=["python3 /greengrass/v2/packages/artifacts/com.example.publish/1.0.0/publish.py..."]}
2022-08-26T13:36:46.237Z [WARN] (Copier) com.example.publish: stderr. python3: can't open file '/greengrass/v2/packages/artifacts/com.example.publish/1.0.0/publish.py': [Errno 2] No such file or directory. {scriptName=services.com.example.publish.lifecycle.Run, serviceName=com.example.publish, currentState=RUNNING}
2022-08-26T13:36:46.252Z [INFO] (Copier) com.example.publish: Run script exited. {exitCode=2, serviceName=com.example.publish, currentState=RUNNING}
```

I can't find anywhere to understand the errors, what is normal and what is not. IT seems like some of these are just saying that publish.py is already running. There is a LOT of AWS documentation. In spite of that documentation I just feel like I am casting about hacking away on hunches feeling my way in the dark. In my opinion the publish tutorial omits something because you can't follow it and get a good result.  Super frustrated here. Not getting any closer.

Answer

Hi flycast.

> I tried installing awsiot on the pi directly but that did not solve the issue.

It's probably because Greengrass is running as a different user than what you installed the package for.

https://docs.aws.amazon.com/greengrass/v2/developerguide/develop-greengrass-components.html

> The AWS IoT Greengrass Core software runs components as the system user and group, such as ggc_user and ggc_group, that you configure on the core device. This means that components have the permissions of that system user. If you use a system user without a home directory, then components can't use run commands or code that use a home directory. This means that you can't use the pip install some-library --user command to install Python packages for example. If you followed the getting started tutorial to set up your core device, then your system user doesn't have a home directory. For more information about how to configure the user and group that run components, see Configure the user that runs components.

One easy solution is to install the packages during the **Install** lifecycle of the recipe. See this example in the workshop.

https://catalog.us-east-1.prod.workshops.aws/workshops/5ecc2416-f956-4273-b729-d0d30556013f/en-US/chapter5-pubsubipc/10-step1

```
"Lifecycle": {
        "Install": "pip3 install awsiotsdk numpy",
        "Run": "python3 -u {artifacts:path}/example_publisher.py"
}
```

Longer term, you may consider to containerize the component.

Answer

As @greg_b mentioned, the issue is likely that your library get installed in the wrong location. There are two solutions:

The first, as mentioned by Greg, is to install the package via the lifecycle install script. You can use the `-u` flag to install as the current user, but be aware that this might not work in case the install script runs with different privileges than the run or startup scripts. A safer option is to use a virtual environment and install the libraries in the virtual environment (https://docs.python.org/3/library/venv.html). Then, in the run or startup script, you just need to activate the virtual environment. The recommendation is to create the virtual environment in the work folder of the component (eg using via `python -m venv ./venv` in the install script. You can check a recipe showing this here https://github.com/awslabs/aws-greengrass-labs-jupyterlab/blob/main/recipes/aws.greengrass.labs.jupyterlab.yaml. By using virtual environments and by keeping them in the component work folder you are also avoiding possible conflicts between libraries and library versions required by different components.

The second would be to package the necessary libraries with your artifact, but this might require that you create the artifact archive on a machine/container that is equivalent to your target device, as some libraries might be dependent on OS and/or CPU architecture.

Finally, I strongly recommend you add any library dependency in a separate component, which you declare as dependency to the main component. In this way you minimize the downtime of your main component when new versions are deployed. Greengrass stops the component process before executing the install script of the new version, and if you do not have any library installation instruction the install phase will execute faster. The library installation happens only the first time the component is deployed to the device and every time a new version of the dependency component is available.

Cheers,
Massimiliano

Answer

Thank you both for answering my questions. This is what I LOVE about programming today vs 1976.

> Finally, I strongly recommend you add any library dependency in a separate component, which you declare as dependency to the main component. In this way you minimize the downtime of your main component when new versions are deployed. Greengrass stops the component process before executing the install script of the new version, and if you do not have any library installation instruction the install will execute faster. The library installation happens only the first time the component is deployed to the device and every time a new version of the dependency component is available.

That all makes sense to me. How is this done though? In this instance I need awsiotsdk correct? If it is needed as a component I am surprised it isn't provided already by AWS. At any rate, how would I go about creating a component of a dependency and then using it in a different private component?

Answer

Hi Flycast,

you can check how dependencies are setup in [this component recipe](https://github.com/awslabs/aws-greengrass-labs-nodered/blob/main/recipe.yaml). At line 12 there is a dependency on [another private component](https://github.com/awslabs/aws-greengrass-labs-nodered-auth/blob/main/recipe.yaml), which also specifies the acceptable versions using semantic versioning notation (if you are not familiar with it, you can try out the notation here https://semver.npmjs.com/) . You can also specify the `DependencyType` as `HARD` or `SOFT`. Hard dependencies **cause** the main component to restart in case they are updated, Soft dependencies don't and will be used only upon restart of the main component (see the [documentation](https://docs.aws.amazon.com/greengrass/v2/developerguide/component-recipe-reference.html#recipe-format)).

When using private components as dependencies make sure you have created the dependent component in the same account and region of the main component before deploying the main component, or the deployment will fail.

You can find more examples of ready to use, curated private components in the [Greengrass Software Catalog](https://github.com/aws-greengrass/aws-greengrass-software-catalog)

Cheers,

Massimiliano

Creating python greengrass scripts that depend on other packages

Relevant content