How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?

Short description

Python packages that contain compiled code (for example: NumPy and pandas) aren't always compatible with Lambda runtimes by default. If you install these packages using pip, then the packages download and compile a module-name package for the architecture of the local machine. This makes your deployment package incompatible with Lambda if you're not using a Linux operating system.

To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with manylinux2014 as the value for the --platform parameter.

Note: macOS --platform tags don't work. For example: The win_amd64 and macosx_10_6_intel tags won't install a deployment package that's compatible with Lambda.

Resolution

Note: This example procedure shows how to install pandas for the Lambda Python 3.9 runtime that's running on x86_64 architecture.

1.    Open a command prompt. Then, confirm that you're using a version of pip that's version 19.3.0 or newer by running the following pip command:

pip --version

If you're using a version of pip that's older than pip version 19.3.0, then upgrade to the latest version of pip by running the following command:

python3.9 -m pip install --upgrade pip

2.    Install the precompiled Python package's .whl file as a dependency in your Lambda function's project directory by running the following command:

Important: Replace my-lambda-function with the name of your function's project directory.

pip install \
    --platform manylinux2014_x86_64 \
    --target=my-lambda-function \
    --implementation cp \
    --python 3.9 \
    --only-binary=:all: --upgrade \
    pandas

3.    Open your Lambda function's project directory. If you're using macOS, then run the following command:

cd my-lambda-function

4.    In a text editor, create a new file named lambda_function.py. Then, copy and paste the following example code into the file and save it in your Lambda function's project directory:

import numpy as np
import pandas as pd
def lambda_handler(event, context):
    df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),columns=["a", "b", "c"])
    number = np.pi
    print(df2)
    print(number)

5.    Create a Lambda deployment package .zip file archive that includes all of the installed libraries and source code by running the following command:

zip -r ../my-deployment-package.zip .

6.    Use the my-deployment-package.zip file archive to either create a new Python 3.9 Lambda function or to update an existing one. For instructions, see Deploy your .zip file to the function in the AWS Lambda Developer Guide.

Note: You can use a similar procedure to create a Lambda layer that can be used across multiple functions. For example, the following command creates a new Lambda layer to install pandas for the Lambda Python 3.9 runtime, running on arm64 architecture:

pip install \
    --platform manylinux2014_aarch64 \
    --target=./python/lib/python3.9/site-packages \
    --implementation cp \
    --python 3.9 \
    --only-binary=:all: --upgrade \
    pandas

Related information

Deploy Python Lambda functions with .zip file archives

Creating and sharing Lambda layers