Lambda Package Exceeds 60MB: Solutions for Large Dependencies?

1. Optimize Your Dependencies

Exclude Unnecessary Packages: Review your requirements.txt or equivalent and remove any unnecessary packages or dependencies. Use Lambda-Compatible Packages: Check if smaller, Lambda-compatible versions of packages are available.

Custom Build: Rebuild some packages from source, removing unnecessary components or dependencies to reduce size.

2. Create Multiple Lambda Layers

Divide Dependencies Across Layers: Instead of one large Lambda Layer, split your dependencies into multiple layers. AWS Lambda allows up to 5 layers to be attached to a single function. You can distribute your packages across these layers to stay within the size limits.

3. Use AWS Lambda with Docker (Container Images)

Switch to Lambda Containers: AWS Lambda supports container images up to 10GB. You can package your Lambda function along with all dependencies into a Docker container image. This approach bypasses the 60MB limit and allows for larger dependencies.

Steps:

Create a Dockerfile with your function code and dependencies.

Build and test the Docker image locally.

Push the image to Amazon Elastic Container Registry (ECR).

Create or update the Lambda function to use the container image from ECR.

Example Dockerfile:

FROM public.ecr.aws/lambda/python:3.9

# Install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy function code
COPY app.py .

# Command to run your Lambda function
CMD ["app.lambda_handler"]

Building and Deploying:

docker build -t my-lambda-function .

docker tag my-lambda-function:latest <your-ecr-repo>:latest

docker push <your-ecr-repo>:latest

4. Load Dependencies Dynamically

Use S3 for Large Packages: Store your large packages in S3 and download them dynamically within your Lambda function at runtime. While this increases cold start time, it can help you stay within the deployment package limits.

**Download and Extract: ** Download and extract dependencies to the /tmp directory in the Lambda function, which provides 512MB of space.

import boto3
import zipfile
import os

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    s3.download_file('my-bucket', 'large-package.zip', '/tmp/large-package.zip')

    with zipfile.ZipFile('/tmp/large-package.zip', 'r') as zip_ref:
        zip_ref.extractall('/tmp/dependencies')

    sys.path.insert(0, '/tmp/dependencies')

    # Your function code

5. Refactor Code to Microservices

Break Down Functionality: If possible, break down your Lambda function into smaller microservices. Each smaller function can handle specific tasks, reducing the size of the deployment package.

6. Use AWS Step Functions

**Orchestrate with Step Functions: **If your large package is due to multiple distinct tasks, consider using AWS Step Functions to orchestrate multiple smaller Lambda functions. This way, each Lambda function can remain within the size limits.

7. Explore Third-Party Solutions

Serverless Framework: Some tools, like the Serverless Framework, offer plugins to help manage large packages and dependencies more effectively.

Summary:

Optimize dependencies to reduce the package size.

Use multiple layers or Docker containers for larger packages.

Dynamically load dependencies from S3 if necessary.

Break down functionality into smaller Lambda functions or microservices.