Unable to import lxml etree on aws lambda

There are modules that cannot be added directly into the site-packages directory to be recognised inside an AWS Lambda environment. When that happens, you have to get an Amazon Linux image from Docker repositories and make your own compiled environment in a container version that will run on AWS Lambda

For example, if you want to use Python 3.6 a good choice will be amazonlinux:2018.03 in case you want to install more packages e.g. pandas, numpy, scipy

docker run -v $(pwd):/outputs -it amazonlinux:2018.03  

Since Amazon Linux is based on Red Hat, you have to install via yum all dependencies when running docker and having already created your virtual environment

yum update -y  
yum install -y \  
  python36 \  
  python36-devel \  
  python36-virtualenv \  
  python34-setuptools \  
  gcc \  
  gcc-c++ \  
  findutils \  
  rsync \  
  Cython \  
  findutils \  
  which \  
  gzip \  
  tar \  
  man-pages \  
  man \  
  wget \  
  make \  
  zip  

For lxml you will also need

(lambda_docker) bash-4.2# yum install libxml2  
...  
(lambda_docker) bash-4.2# yum install libxslt  
...  

You install the module as usual

pip3.6 install lxml  

You should see something like

(lambda_docker) bash-4.2# pip3.6 install lxml  
Collecting lxml  
  Downloading https://files.pythonhosted.org/packages/2d/53/34a9f0c79c548e430148837892b6ae91adee571a0e8b6c17bd7ff9c2d12e/lxml-4.3.4-cp36-cp36m-manylinux1_x86_64.whl (5.7MB)  
     |################################| 5.7MB 2.0MB/s   
Installing collected packages: lxml  

Then, create your function lambda_function.py and add it in the zipped package pushd-ed and popd-ed from you docker session

from lxml import etree  

def lambda_handler(event, context):  
    print(__name__)  
    print(etree.LXML_VERSION)  

After created

zip -g site-packages.zip lambda_function.py  

Before uploading, you can check that your zip file contains the lxml directories

\[jonathan@docker lambda_docker]$ unzip -l site-packages.zip   
Archive:  site-packages.zip  
  Length      Date    Time    Name  
---------  ---------- -----   ----  
        0  06-29-2019 23:09   __pycache__/  
      277  06-29-2019 23:09   __pycache__/easy_install.cpython-36.pyc  
      126  06-29-2019 23:09   easy_install.py  
      119  06-29-2019 23:29   lambda_function.py  
        0  06-29-2019 23:21   lib/  
        0  06-29-2019 23:39   lxml/  
        0  06-29-2019 23:37   lxml-4.3.4.dist-info/  
        4  06-29-2019 23:37   lxml-4.3.4.dist-info/INSTALLER  
     2954  06-29-2019 23:37   lxml-4.3.4.dist-info/METADATA  
    13384  06-29-2019 23:37   lxml-4.3.4.dist-info/RECORD  
      109  06-29-2019 23:37   lxml-4.3.4.dist-info/WHEEL  
        5  06-29-2019 23:37   lxml-4.3.4.dist-info/top_level.txt  
     7668  06-29-2019 23:37   lxml/ElementInclude.py  
      551  06-29-2019 23:37   lxml/__init__.py  
        0  06-29-2019 23:37   lxml/__pycache__/  
     3331  06-29-2019 23:37   lxml/__pycache__/ElementInclude.cpython-36.pyc  
...  

Now upload the zip and create an empty test {} in your lambda function

Result

START RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64 Version: $LATEST  
lambda_function  
(4, 3, 4, 0)  
END RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64  
REPORT RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64  Duration: 0.30 ms   Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 50 MB