Unable to import lxml etree on aws lambda

Question

{  
  "errorMessage": "Unable to import module 'lambda_function':   
        cannot import name   'etree' from 'lxml' (/var/task/lxml/__init__.py)",  
  "errorType": "Runtime.ImportModuleError"  
}  
Also tried https://gist.github.com/allen-munsch/ad8faf9c04b72aa8d0808fa8953bc639:  
  
{  
  "errorMessage": "Unable to import module 'lambda_function':   
     cannot import name 'etree' from 'lxml'   
     (/var/task/lxml-4.3.4-py3.6-linux-x86_64.egg/lxml/__init__.py)",  
  "errorType": "Runtime.ImportModuleError"  
}  
I am running on Ubuntu 18.04 on my local machine, and have also tried using the "Amazon Linux" image on an ec2 instance to build the bundle.  
  
I have also tried, while within the activated venv:  
  
STATIC_DEPS=true pip3 install lxml --target ./package --upgrade --no-cache-dir  
I have also tried copying the shared object files based on pulling which files were opened when running the script with strace:  
  
#! /bin/bash  
  
export Z=$(pwd)/ok-daily-lambda.zip  
rm $Z  
zip $Z lambda_function.py  
zip $Z __init__.py  
  
for dir in $(find venv_here/lib/python3.6/site-packages)  
do  
    if \[ -d $dir ] ; then  
        pushd $dir  
        echo zip -r9 $Z $(pwd)  
        zip -r9 $Z $(pwd)  
        popd  
    fi  
done  
  
export LIBD=$(pwd)/lib  
mkdir -p $LIBD  
  
cp "/home/jmunsch/.local/lib/python3.6/site-packages/.libs_cffi_backend/libffi-d78936b1.so.6.0.4" $LIBD  
cp "/lib/x86_64-linux-gnu/libbz2.so.1.0" $LIBD  
cp "/lib/x86_64-linux-gnu/libc.so.6" $LIBD  
cp "/lib/x86_64-linux-gnu/libdl.so.2" $LIBD  
cp "/lib/x86_64-linux-gnu/libexpat.so.1" $LIBD  
cp "/lib/x86_64-linux-gnu/libgcc_s.so.1" $LIBD  
cp "/lib/x86_64-linux-gnu/liblzma.so.5" $LIBD  
cp "/lib/x86_64-linux-gnu/libm.so.6" $LIBD  
cp "/lib/x86_64-linux-gnu/libnss_dns.so.2" $LIBD  
cp "/lib/x86_64-linux-gnu/libnss_files.so.2" $LIBD  
cp "/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2" $LIBD  
cp "/lib/x86_64-linux-gnu/libpthread.so.0" $LIBD  
cp "/lib/x86_64-linux-gnu/libresolv.so.2" $LIBD  
cp "/lib/x86_64-linux-gnu/librt.so.1" $LIBD  
cp "/lib/x86_64-linux-gnu/libtinfo.so.5" $LIBD  
cp "/lib/x86_64-linux-gnu/libudev.so.1" $LIBD  
cp "/lib/x86_64-linux-gnu/libutil.so.1" $LIBD  
cp "/lib/x86_64-linux-gnu/libuuid.so.1" $LIBD  
cp "/lib/x86_64-linux-gnu/libz.so.1" $LIBD  
cp "/usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0" $LIBD  
cp "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1" $LIBD  
cp "/usr/lib/x86_64-linux-gnu/libffi.so.6" $LIBD  
cp "/usr/lib/x86_64-linux-gnu/liblz4.so.1" $LIBD  
cp "/usr/lib/x86_64-linux-gnu/libmpdec.so.2" $LIBD  
cp "/usr/lib/x86_64-linux-gnu/libssl.so.1.1" $LIBD  
cp "/usr/lib/x86_64-linux-gnu/libstdc++.so.6" $LIBD  
cp "/usr/lib/x86_64-linux-gnu/libzstd.so.1" $LIBD  
  
zip -r $Z $LIBD  
  
AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="xxx" AWS_DEFAULT_REGION="us-east-1" aws lambda update-function-code --function-name ok-today --zip-file fileb://ok-daily-lambda.zip  
Here is the directory structure of the most recent zip file:  
  
https://gist.github.com/allen-munsch/5ddc27873e64db6ff106ab828febf434

KalindiP · Answer

Hi, 
This might be a little late to answer, but here is an alternative approach that helped me fixed this issue without spinning an ec2  instance. 
For reference, I was getting the same issue with my Lambda function when I was using the python-docx library.

1) The easiest way is to utilize your cloud shell environment. The AWs cloud shell had a similar runtime environment as my python lambda. You can pip install your libraries, package them as you would traditionally do for a normal zip file and then the AWS cloud shell has an option to download the zip file to your local desktop. You can download it and then upload the zip file to your lambda layer and use it as a dependency.  Worked well for me and saves me the hassle of spinning a VM every time.

Few things to note would be:
1) The python version on your cloud shell might not be the same as the version on your lambda, but it very easy to upgrade/downgrade it on your cloud shell before you use it. 
2) The initial zip file that I created had a folder named python-3.12.x but it did not work with that, but renaming it 3.12 worked.

Hope this helps!

Unable to import lxml etree on aws lambda

Add your answer

Relevant content