Using Lambda for data processing - Sagemaker

0

I have created a docker image which has Entrypoint as processing.py. This script is taking data from /opt/ml/processing/input and after processing putting it /opt/ml/processing/output folder.

For processing the data I should put the file in /opt/ml/processing/input from s3 and then pick processed file from /opt/ml/processing/output into S3.

Following script in sagemaker is doing it properly:

from sagemaker.processing import Processor, ProcessingInput, ProcessingOutput import sagemaker

input_data = 's3://sagemaker-ap-south-1-057036842446/sagemaker/Data/Training/Churn_Modelling.csv' output_dir = 's3://sagemaker-ap-south-1-057036842446/sagemaker/Outputs/' image_uri = '057036842446.dkr.ecr.ap-south-1.amazonaws.com/aws-docker-repo:latest' aws_role = sagemaker.get_execution_role()

processor = Processor(image_uri= image_uri, role=aws_role, instance_count=1, instance_type="ml.m5.xlarge")

processor.run(inputs=[ProcessingInput( source=input_data, destination='/opt/ml/processing/input')], outputs=[ProcessingOutput(source='/opt/ml/processing/output', destination=output_dir)] )

Could someone please guide how this can be executed with lambda function? It is not recognizing sagemaker package, second there is a challenge in placing file before the script execution and pick processed files.

I am trying codepipeline to automate this operation.

2 Answers
1

SageMaker Python SDK, as a third party Python package, is not naturally installed Lambda's default runtime. You need to install this package.

You can either follow steps in https://docs.aws.amazon.com/lambda/latest/dg/python-package.html to form a zip archive or similarly install sagemaker Python SDK as a layer as in https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

This should provide a similar runtime env as you container.

AWS
answered a year ago
0

I have build a codepipeline where after build I am executing lambda in the deploy stage.

In Lamdba I am executing the docker image.

To execute docker image I need to first provide it file from s3 and after script processing file generated need to move to S3.

Is it actually possible in Lambda limitations?

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions