unable to import lxml into python lambda: cannot import name 'etree' from 'lxml'

Question

While trying to update my lambda from 3.8 to **3.12 ** I faced issues on the imports of the lxml package.

Runtime.ImportModuleError: Unable to import module 'lambda_function': cannot import name 'etree' from 'lxml' (/opt/python/lxml/__init__.py)

I tried to follow the solution described in https://repost.aws/questions/QUwIZg6DlXTgKHjJyMIlG1dQ/unable-to-import-lxml-etree-on-aws-lambda. However, this was for an older version and didn't seem to work.
I used the following dockerfile to generate the deployement zip file for the layer:

`FROM amazonlinux
RUN yum update -y && yum install -y \
      gcc \
      gcc-c++ \
      make \
      tar \
      gzip \
      wget \
      findutils \
      rsync \
      which \
      zip \
      man-pages \
      man \
      python3.12 \
      python3.12-devel \
      python3-pip \
      Cython \
      libxml2 \
      libxslt

WORKDIR /app

COPY requirements.txt .
RUN mkdir python
RUN pip3 install --target=python -r requirements.txt
RUN zip -r layer.zip python`

**My lambda has the following requirements:**
qrbill
svglib
reportlab

where lxml seems to be an included dependency so I also included it into my requirements.txt file.

What could be the problem? Is there an easy fix without using a EC2 environment or other complex solutions?

Riku_Kobayashi · Accepted Answer

Hello.

How about creating a layer using CloudShell as shown below?  
CloudShell runs on an Amazon Linux 2023-based container, so I think OS-level dependencies can be resolved.  
```
sudo dnf install gcc zlib-devel bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel xz-devel -y
curl https://pyenv.run | bash
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
. .bashrc
pyenv install 3.12.8
pyenv global 3.12.8
python --version

LIB_DIR=mylayer/python
mkdir -p $LIB_DIR
pip3 install qrbill -t $LIB_DIR
pip3 install svglib -t $LIB_DIR
pip3 install reportlab -t $LIB_DIR
cd mylayer
zip -r /tmp/mylayer.zip .
aws lambda publish-layer-version --layer-name mylayer --zip-file fileb:///tmp/mylayer.zip --compatible-runtimes python3.12
aws lambda update-function-configuration --function-name  --layers arn:aws:lambda:::layer:mylayer:1
```

I have confirmed that I can create a layer using the above command in my AWS account and import it using the Lambda function below.  
```
from lxml import etree

def lambda_handler(event, context):  
    print(__name__)  
    print(etree.LXML_VERSION)
```

Please refer to the following document for how to use CloudShell.  
https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html

re:Post Agent · Answer

The issue you're experiencing with importing lxml in your AWS Lambda environment is likely due to compatibility problems between the compiled binary dependencies and the Lambda execution environment. This is a common challenge when working with libraries that have C or C++ components, like lxml.

To resolve this issue, you have a few options:

1. Use a Lambda Layer: Create a Lambda Layer containing the lxml library and its dependencies, compiled specifically for the Lambda environment. This approach allows you to separate the library from your function code and reuse it across multiple functions.

2. Use a container image: Instead of packaging your function as a ZIP file, you can create a container image that includes the correct version of lxml and its dependencies. This gives you more control over the runtime environment.

3. Build in a compatible environment: Use an Amazon Linux environment that matches the Lambda execution environment to build your deployment package. This ensures that the compiled binaries are compatible.

4. Use AWS SAM CLI with --use-container option: If you're using AWS SAM for deployment, you can use the --use-container option when running 'sam build'. This builds your function's deployment package in a Docker container using a compatible base image.

5. Manually build the .zip deployment package: Build your function's .zip deployment package yourself using the correct source distribution binary for lxml.

For Python 3.12 in Lambda, which is based on Amazon Linux 2023 (AL2023), you might need to ensure you're using a compatible version of lxml. You could try specifying a version in your requirements.txt file that is known to work with Python 3.12 and AL2023.

If you want to avoid using an EC2 environment or other complex solutions, using a Lambda Layer or a container image might be your best options. These approaches allow you to package the library and its dependencies in a way that's compatible with the Lambda environment without needing to set up a separate build environment.

Remember to thoroughly test your function after making these changes to ensure that the lxml library is working correctly in the Lambda environment.  
**Sources**  
[Lambda execution environment has older version of GLIBC | AWS re:Post](https://repost.aws/questions/QU1OW0AWrmRH-IIzk-zSE9zA/lambda-execution-environment-has-older-version-of-glibc)  
[Deploy Python Lambda functions with container images - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/python-image.html)  
[Issue with Importing Pillow Library for Image Processing in AWS Lambda Environment | AWS re:Post](https://repost.aws/questions/QU11QL_JaISAOSykJteHyFHg/issue-with-importing-pillow-library-for-image-processing-in-aws-lambda-environment)

unable to import lxml into python lambda: cannot import name 'etree' from 'lxml'

Add your answer

Relevant content