AWS Lambda Layers with Python Package

0

Hi,

I have a python script which is running on my local machine and I want to move it to AWS Lambda for periodic execution. I have 3 import statements in the script for which I am adding layers but facing some issues.

from googleapiclient.discovery import build

import pandas as pd

from datetime import date

For googleapiclient api, I downloaded it in a folder and uploaded to AWS layer and lambda is able to find this module. I wanted to use this along with AWS Data wrangler package but running into layer size restriction issues. So I downloaded pandas to the same folder as google api and then uploaded the zip file to layer. But now I am get a numpy dependency error though numpy was downloaded as part of pandas install. Two folders as part of my libraries folder are numpy, numpy-1.22.0.dist-info which is correct version as per error message below. I also tried downloading numpy separately in same package but that also not working.

Please let me know if I am missing something and if this is the correct approach for installing python packages for AWS Lambda.

Response { "errorMessage": "Unable to import module 'lambda_function': Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from "/var/lang/bin/python3.7"\n * The NumPy version is: "1.22.0"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n", "errorType": "Runtime.ImportModuleError", "stackTrace": [] }

Function Logs START RequestId: de7259a6-cd11-4ecc-9f93-1a624f1c0c6d Version: $LATEST [ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': Unable to import required dependencies: numpy: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following:

  • The Python version is: Python3.7 from "/var/lang/bin/python3.7"
  • The NumPy version is: "1.22.0" and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help. Original error was: No module named 'numpy.core._multiarray_umath' Traceback (most recent call last): END RequestId: de7259a6-cd11-4ecc-9f93-1a624f1c0c6d REPORT RequestId: de7259a6-cd11-4ecc-9f93-1a624f1c0c6d Duration: 1.68 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 55 MB Init Duration: 570.09 ms

Request ID de7259a6-cd11-4ecc-9f93-1a624f1c0c6d

Regards, Dbeings

dbeing
asked 2 years ago3697 views
1 Answer
3

As the error states the issue with how you NumPy was installed. Most probably you compiled it on your dev machines which might be windows, mac or some flavor of linux. While lambda runs on Amazon Linux. Since NumPy is a C extension it is better that you build the Python package for your panda and NumPy using Amazon Linux.

You can easily use a docker image on your dev machine to build the packages or you can deploy an EC2 instance to build the package as well and then terminate the ec2 instance. If you use CodeBuild to build your packages then CodeBuild also runs on Amazon Linux.

answered 2 years ago
  • Thanks a ton! This was indeed the issue. I now used EC2 instance to install pandas files in a folder ( pip install pandas -t .) and then copied the files to my machine and then put the zip folder on my AWS layer. This worked.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions