Lambda can not load python modules installed with component

Question

Hi,

I have installed Greengrass Core on a Raspberry Pi 4.
I want to run a python script using Lambda functions. For that I created 2 components.
ComponentA installs all python dependencies required to run python script.
ComponentB is the Lambda component that will run the python script.

When I deploy both components I can see that ComponentA installs the python packages under gcc_user user directory.
But ComponentB always fails because it can not find/load the packages required.

Both components should be run as the same gcc_user. Any ideas?

Bests,

ComponentA recipe :
{
"RecipeFormatVersion": "2020-01-25",
"ComponentName": "componentA",
"ComponentVersion": "1.0.3",
"ComponentType": "aws.greengrass.generic",
"ComponentDescription": "DHT11 dependencies",
"ComponentPublisher": "SomePublisher",
"ComponentConfiguration": {},
"Manifests": [
{
"Platform": {
"os": "linux"
},
"Name": "Linux",
"Lifecycle": {
"Install": "python3 -m pip install dht11 RPi.GPIO boto3"
},
"Artifacts": []
}
],
"Lifecycle": {}
}

Logs for ComponentA:
ComponentA stdout. Downloading https://files.pythonhosted.org/packages/ea/43/4b4a1b26eb03a429a4c37ca7fdf369d938bd60018fc194e94b8379b0c77c/s3transfer-0.3.4-py2.py3-none-any.whl (69kB). {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Collecting botocore<1.20.0,>=1.19.59 (from boto3). {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Downloading https://files.pythonhosted.org/packages/ee/10/08dc3b74cc9c47a2c81b2e88e06c2661783b86fd77fc80f7a3eb1bf56905/botocore-1.19.59-py2.py3-none-any.whl (7.2MB). {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Collecting urllib3<1.27,>=1.25.4; python_version != "3.4" (from botocore<1.20.0,>=1.19.59->boto3). {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Downloading https://files.pythonhosted.org/packages/f5/71/45d36a8df68f3ebb098d6861b2c017f3d094538c0fb98fa61d4dc43e69b9/urllib3-1.26.2-py2.py3-none-any.whl (136kB). {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.20.0,>=1.19.59->boto3). {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB). {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.20.0,>=1.19.59->boto3) (1.12.0). {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Installing collected packages: jmespath, urllib3, python-dateutil, botocore, s3transfer, boto3. {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}
ComponentA stdout. Successfully installed boto3-1.16.59 botocore-1.19.59 jmespath-0.10.0 python-dateutil-2.8.1 s3transfer-0.3.4 urllib3-1.26.2. {scriptName=services.componentA.lifecycle.Install, serviceName=componentA, currentState=NEW}

ls -ll /home/ggc_user/.local/lib/python3 drwxr-xr-x 4 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group -rwxr-xr-x 1 ggc_user drwxr-xr-x 4 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group drwxr-xr-x 4 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group drwxr-xr-x 10 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group drwxr-xr-x 7 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group drwxr-xr-x 6 ggc_user ggc_group drwxr-xr-x 3 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group drwxr-xr-x 3 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group drwxr-xr-x 3 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group drwxr-xr-x 6 ggc_user ggc_group drwxr-xr-x 2 ggc_user ggc_group .7/site-packages
4096 Jan 6 12:52 RPi
4096 Jan 6 12:52 RPi.GPIO-0.7.0.dist-info
ggc_group 10834284 Jan 5 11:33 _awscrt.cpython-37m-arm-linux-gnueabihf.so
4096 Jan 5 11:33 awscrt
4096 Jan 5 11:33 awscrt-0.9.15.dist-info
4096 Jan 5 11:33 awsiot
4096 Jan 5 11:33 awsiotsdk-1.5.3.dist-info
4096 Jan 26 10:31 boto3
4096 Jan 26 10:31 boto3-1.16.59.dist-info
4096 Jan 26 10:31 botocore
4096 Jan 26 10:31 botocore-1.19.59.dist-info
4096 Jan 26 10:31 dateutil
4096 Jan 6 12:52 dht11
4096 Jan 6 12:52 dht11-0.1.0.dist-info
4096 Jan 26 10:31 jmespath
4096 Jan 26 10:31 jmespath-0.10.0.dist-info
4096 Jan 26 10:31 python_dateutil-2.8.1.dist-info
4096 Jan 26 10:31 s3transfer
4096 Jan 26 10:31 s3transfer-0.3.4.dist-info
4096 Jan 26 10:31 urllib3
4096 Jan 26 10:31 urllib3-1.26.2.dist-info

Logs for ComponentB:
ComponentB: FATAL: lambda_runtime.py:147,Failed to import handler function "lambda_function.lambda_handler" due to exception: No module named 'boto3'. {serviceInstance=0, serviceName=componentB, currentState=RUNNING}
ComponentB: FATAL: lambda_runtime.py:428,Failed to initialize Lambda runtime due to exception: No module named 'boto3'. {serviceInstance=0, serviceName=componentB, currentState=RUNNING}
ComponentB: shell-runner-start. {scriptName=services.componentB.lifecycle.shutdown.script, serviceInstance=0, serviceName=componentB, currentState=BROKEN, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.3/lambda-l..."]}

Answer

The component which is trying to use these Python libraries is a lambda? Is the lambda running in container mode or no container mode?  
  
You may try installing the Python modules globally instead of user-specific which should let the lambda mount them into the container.  
  
Cheers,  
Michael Dombrowski

Answer

Hi Michael,  
  
The component which is trying to use these Python libraries is a lambda function. This Lambda function is running in "no container mode".  
  
If I ssh into core device and install the modules globally (as root user) it works as expected but this is not what I want. In a real world environment I might not able to ssh into the core device.  
  
Installing dependencies globally requires root privileges and Lambda functions are run as ggc_user as default. So It seems impossible to install the dependencies globally without connecting the device. Besides, it should work as it is I guess.  
  
Any other suggestions?  
  
Cheers!

Answer

Hi Michael,  
  
Thanks for the answer.  
Giving a Lambda root access and installing dependencies globally is not an option for us.  
I tried setting PYTHONPATH environment variable to the user python package location but Python search paths got messed up totally.  
  
I found a very clean method to import a python path on the fly.  
I finally got it working using "site" python module.  
At the beginning of the Lambda function I used below code and all the packages installed from another lambda got resolved.  
  
import site  
site.addsitedir('/home/ggc_user/.local/lib/python3.7/site-packages')   
  
Bests

Answer

You can globally install the modules with privileges by telling Greengrass to run the install step as root. See https://docs.aws.amazon.com/greengrass/v2/developerguide/component-recipe-reference.html#recipe-format, look for "RequiresPrivilege".  
  
The likely reason why it doesn't load the user's path is because lambdas set the PYTHONPATH environment variable so that local python files will work, however I believe that Python will then not load from the users's path. You can address this by not using a lambda or by setting the PYTHONPATH environment variable in the lambda's recipe so that it contains the user home path. The last option, as I discussed earlier, is to install globally using RequiresPrivilege.

Answer

Are you not able to use the site package, and I assume you are not blocked?

Answer

The only thing that worked for me was:  
sudo -H -u ggc_user python3 -m pip install your_module

Lambda can not load python modules installed with component

Relevant content