Using Lambda for data processing - Sagemaker

Question

I have created a docker image which has Entrypoint as processing.py. This script is taking data from /opt/ml/processing/input and after processing putting it /opt/ml/processing/output folder.

For processing the data I should put the file in /opt/ml/processing/input from s3 and then pick processed file from /opt/ml/processing/output into S3.

Following script in sagemaker is doing it properly:

from sagemaker.processing import Processor, ProcessingInput, ProcessingOutput
import sagemaker

input_data = 's3://sagemaker-ap-south-1-057036842446/sagemaker/Data/Training/Churn_Modelling.csv'
output_dir = 's3://sagemaker-ap-south-1-057036842446/sagemaker/Outputs/'
image_uri = '057036842446.dkr.ecr.ap-south-1.amazonaws.com/aws-docker-repo:latest'
aws_role = sagemaker.get_execution_role()

processor = Processor(image_uri= image_uri,
                     role=aws_role,
                     instance_count=1,
                     instance_type="ml.m5.xlarge")

processor.run(inputs=[ProcessingInput(
                        source=input_data,
                        destination='/opt/ml/processing/input')],
             outputs=[ProcessingOutput(source='/opt/ml/processing/output',
                                       destination=output_dir)]
                    )

Could someone please guide how this can be executed with lambda function? It is not recognizing sagemaker package, second there is a challenge in placing file before the script execution and pick processed files.

I am trying codepipeline to automate this operation.

Answer

SageMaker Python SDK, as a third party Python package, is not naturally installed Lambda's default runtime. You need to install this package.

You can either follow steps in https://docs.aws.amazon.com/lambda/latest/dg/python-package.html to form a zip archive or similarly install sagemaker Python SDK as a layer as in https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

This should provide a similar runtime env as you container.

Answer

I have build a codepipeline where after build I am executing lambda in the deploy stage.

In Lamdba I am executing the docker image.

To execute docker image I need to first provide it file from s3 and after script processing file generated need to move to S3.

Is it actually possible in Lambda limitations?

Using Lambda for data processing - Sagemaker

Contenido relevante