By using AWS re:Post, you agree to the Terms of Use

AWS Greengrass lambda component hangs on python import of GreengrassCoreIPCClientV2

0

Hello, I am in need of some troubleshooting help.

I am running Greengrassv2 and are building a lambda component. The target runtime (greengrass nucleus) runs on a raspberry pi (arm32v7). Hence I am creating the lambda deployment package in a container based on arm32v7/python:3.9-slim-bullseye. My lambda deployment package contains e.g. "_awscrt.cpython-39-arm-linux-gnueabihf.so" which makes me think I have made a correct compilation of the wheels.

Now to the issue I am facing My lambda function component works as expected until I do an import such as from awsiot.greengrasscoreipc.clientv2 import GreengrassCoreIPCClientV2. When I do that the function just hangs, without errors, I have a debug printout before the import that is displayed correctly in the component log.

I am running the latest version of all components, e.g. 2.7.0 of greengrass nucleus.

The component configuration states "containerMode": "GreengrassContainer".

I do not yet have that much debug information, since the lambda-component log is quiet and the and greengrass.log is happy pappy, its latest row just states that is has published to Local PubSub topic (which triggered my Lambda component).

If I execute the lambda function in AWS I instantly receive an error: "No module named 'pycares._cares'". <---- which can be expected since the wheels are not compiled for amazon linux2 arch.

I am suspecting something about python version, I have yet to check which python version is used by greengrass at runtime. However the raspbery pi is using pyenv to set python 3.8.3 globally. And as mentioned I have compiled the lambda deployment package using python3.9. This just struck me so I will investigate this asap.

Any troubleshooting tips are very welcome.

2 Answers
2
Accepted Answer

Hi @rikerik

I would highly recommend creating a Greengrass v2 custom component instead of a Greengrass Lambda to run your code on Greengrass. In Greengrass V2, Lambdas are supported for backward compatibility to Greengrass v1.

To get you started, take a look at this tutorial, Step 5 of the tutorial has step-by-step instructions to create a simple python based Hello World component.

Greengrass v2 also provides a Greengrass Development Kit CLI that simplifies the creation and deployment of components to AWS Greengrass.

Compared to Lambdas, Greengrass components run as an OS process and not in a container. This makes dependency management easier. Additionally, you can add setup commands to install dependencies in the Greengrass component recipe.

Greengrass Components can also subscribe to local PubSub topics, a python code sample can be found here

Hope that helps

profile picture
answered 15 days ago
  • Thank you, migrating to a custom component would definably have solved my issue since it had to do with lack of working memory in the container.

    Since I am using serverless framework to build and deploy the lambda function I refrained from immediately switching over to building a custom component. Is it your opinion that lambdas will be deprecated in the future of GGv2?

    And a question, you wrote: "Additionally, you can add setup commands to install dependencies in the Greengrass component recipe." in your answer.

    • Are you by that recommending that I should do "pip install" like that and by that have the wheels compiled correctly since it would execute on the target OS?
  • I would still recommend creating a virtual environment per component to isolate the dependencies, here a sample recipe :

    ...
    Manifests:
      - Platform:
          os: /linux|darwin/
        Lifecycle:
          Install: |-
            python3 -m venv venv
            . venv/bin/activate
            pip3 install --upgrade pip
            pip3 install wheel
            pip3 install awsiotsdk
    Run: 
            Script: |-
              . venv/bin/activate
              python3 -u {artifacts:path}/mycomponent.py
    
1

Hi,

The reason why this is happening is that importing those additional modules is putting your application above the memory limit and then killing it. The memory limit is configured when you create the component from a lambda, the default is 16MB. You can address this by using "NoContainer" mode instead of a container. You may also increase the memory limit or simply use a native component as suggested by @Jan_B.

If you do change the memory limit or container mode, make sure that you use "RESET": [""] in the deployment configuration update for the component such that the new default values will be used instead of the existing values on the device. https://docs.aws.amazon.com/greengrass/v2/developerguide/update-component-configurations.html#reset-configuration-update.

Cheers, Michael

answered 12 days ago
  • Thank you!

    You were spot on, the container did indeed run out of memory. I upped the lambda component to 32MB and I am now able to import the GreengrassCoreIPCClientV2 class.

    I guess this means that the container is running armv7l just as the host OS, otherwise the import should have failed, right?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions