using pynini (or openFST) package in serverless lambda

Question

Hello all,
I'm trying to deploy a serverless lambda that uses pynini python library (it has an import statement of it). I added the package to the requirement/base file but the deploy keeps failing due to problems with fst installation. 
I tried to add the bin and lib files of the fst library as well, and add this statements:

os.environ['LD_LIBRARY_PATH'] = f"{os.environ['LD_LIBRARY_PATH']}:{os.environ['LAMBDA_TASK_ROOT']}/resources/fst/lib"
os.environ['PATH'] = f"{os.environ['PATH']}:{os.environ['LAMBDA_TASK_ROOT']}/resources/fst/bin"

But it doesn't work as well. I get this error during deploy:
extensions/_pywrapfst.cpp:815:10: fatal error: 'fst/util.h' file not found......Could not build wheels for pynini, which is required to install pyproject.toml-based projects

I have this headers files as well I just don't know how to tell the lanbda where they are...
I would appreciate any help or ideas, thanks a lot

Answer

Hello,

Firstly in order to add "pynini" module in your Lambda fucntion, utilising 'Lambda containers' based on a 'non-AWS base image' is a recommended approach. To achieve this, you can create a Dockerfile that leverages a non-Amazon Linux machine and adds the necessary packages to it.

By opting for a non-AWS base image, you gain the flexibility to select any distribution base image. This choice proves valuable in addressing dependency issues associated with the "pynini" module.

For detailed guidance on creating a Lambda container-based function, please refer to the AWS documentation provided at the following link:

[+] https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-image-clients

using pynini (or openFST) package in serverless lambda

Relevant content