By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

AWS Greengrass lambda component hangs on python import of GreengrassCoreIPCClientV2

0

Hello, I am in need of some troubleshooting help.

I am running Greengrassv2 and are building a lambda component. The target runtime (greengrass nucleus) runs on a raspberry pi (arm32v7). Hence I am creating the lambda deployment package in a container based on arm32v7/python:3.9-slim-bullseye. My lambda deployment package contains e.g. "_awscrt.cpython-39-arm-linux-gnueabihf.so" which makes me think I have made a correct compilation of the wheels.

Now to the issue I am facing My lambda function component works as expected until I do an import such as from awsiot.greengrasscoreipc.clientv2 import GreengrassCoreIPCClientV2. When I do that the function just hangs, without errors, I have a debug printout before the import that is displayed correctly in the component log.

I am running the latest version of all components, e.g. 2.7.0 of greengrass nucleus.

The component configuration states "containerMode": "GreengrassContainer".

I do not yet have that much debug information, since the lambda-component log is quiet and the and greengrass.log is happy pappy, its latest row just states that is has published to Local PubSub topic (which triggered my Lambda component).

If I execute the lambda function in AWS I instantly receive an error: "No module named 'pycares._cares'". <---- which can be expected since the wheels are not compiled for amazon linux2 arch.

I am suspecting something about python version, I have yet to check which python version is used by greengrass at runtime. However the raspbery pi is using pyenv to set python 3.8.3 globally. And as mentioned I have compiled the lambda deployment package using python3.9. This just struck me so I will investigate this asap.

Any troubleshooting tips are very welcome.

Topics
Internet of Things (IoT)ServerlessCompute
Tags
AWS IoT GreengrassAWS IoT CoreAWS Lambda
Language
English
rikerik
asked 2 years ago521 views
2 Answers
2
Accepted Answer

Hi @rikerik

I would highly recommend creating a Greengrass v2 custom component instead of a Greengrass Lambda to run your code on Greengrass. In Greengrass V2, Lambdas are supported for backward compatibility to Greengrass v1.

To get you started, take a look at this tutorial, Step 5 of the tutorial has step-by-step instructions to create a simple python based Hello World component.

Greengrass v2 also provides a Greengrass Development Kit CLI that simplifies the creation and deployment of components to AWS Greengrass.

Compared to Lambdas, Greengrass components run as an OS process and not in a container. This makes dependency management easier. Additionally, you can add setup commands to install dependencies in the Greengrass component recipe.

Greengrass Components can also subscribe to local PubSub topics, a python code sample can be found here

Hope that helps

profile pictureAWS
EXPERT
Jan_B
answered 2 years ago
profile picture
EXPERT
Sedat Salman
reviewed 10 months ago
  • rikerik
    2 years ago

    Thank you, migrating to a custom component would definably have solved my issue since it had to do with lack of working memory in the container.

    Since I am using serverless framework to build and deploy the lambda function I refrained from immediately switching over to building a custom component. Is it your opinion that lambdas will be deprecated in the future of GGv2?

    And a question, you wrote: "Additionally, you can add setup commands to install dependencies in the Greengrass component recipe." in your answer.

    • Are you by that recommending that I should do "pip install" like that and by that have the wheels compiled correctly since it would execute on the target OS?
  • Jan_B EXPERT
    2 years ago

    I would still recommend creating a virtual environment per component to isolate the dependencies, here a sample recipe :

    ...
Manifests:
  - Platform:
      os: /linux|darwin/
    Lifecycle:
      Install: |-
        python3 -m venv venv
        . venv/bin/activate
        pip3 install --upgrade pip
        pip3 install wheel
        pip3 install awsiotsdk
Run: 
        Script: |-
          . venv/bin/activate
          python3 -u {artifacts:path}/mycomponent.py
1

Hi,

The reason why this is happening is that importing those additional modules is putting your application above the memory limit and then killing it. The memory limit is configured when you create the component from a lambda, the default is 16MB. You can address this by using "NoContainer" mode instead of a container. You may also increase the memory limit or simply use a native component as suggested by @Jan_B.

If you do change the memory limit or container mode, make sure that you use "RESET": [""] in the deployment configuration update for the component such that the new default values will be used instead of the existing values on the device. https://docs.aws.amazon.com/greengrass/v2/developerguide/update-component-configurations.html#reset-configuration-update.

Cheers, Michael

AWS
EXPERT
MichaelDombrowski-AWS
answered 2 years ago
  • rikerik
    2 years ago

    Thank you!

    You were spot on, the container did indeed run out of memory. I upped the lambda component to 32MB and I am now able to import the GreengrassCoreIPCClientV2 class.

    I guess this means that the container is running armv7l just as the host OS, otherwise the import should have failed, right?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content