SageMaker Multi Model endpoint creation fails while creating for model built on container sagemaker-scikit-learn:0.23-1-cpu-py3


I am working on a use-case where I am using SageMaker multi-model endpoint for model inference and the models are trained using Databricks MLFlow platform. When I tried deploying a model trained from Databricks MLFlow platform on a single endpoint on SageMaker then it worked fine but the creation of multi-model endpoint for 'sagemaker-scikit-learn:0.23-1-cpu-py3' container is failed with the following error:

Code Snippet::>> name = "sample-mme"

sagemaker_client = boto3.client('sagemaker')

model_path = "s3://test-bucket/multi-models"

execution_role_arn = "IAM://sample-role"

BASE_IMAGE = image_uris.retrieve( region=region, framework="sklearn",version='0.23-1',image_scope='inference' )

container = { 'Image': BASE_IMAGE, 'ModelDataUrl': model_path, 'Mode': 'MultiModel', 'MultiModelConfig': { 'ModelCacheSetting': 'Enabled' } }

model_response = sagemaker_client.create_model( ModelName=name, ExecutionRoleArn=execution_role_arn, Containers=[container] )

config_response = sagemaker_client.create_endpoint_config( EndpointConfigName=f'{name}-config', ProductionVariants=[ { 'InstanceType': instance_type, 'InitialInstanceCount': instance_count, 'InitialVariantWeight': 1, 'ModelName': name, 'VariantName': 'AllTraffic' } ] )

response = sagemaker_client.create_endpoint( EndpointName=f'{name}-endpoint', EndpointConfigName=f'{name}-config' )

Endpoint creation is taking a lot if time and failing with the following error message :

sagemaker_containers._errors.ImportModuleError: 'NoneType' object has no attribute 'startswith'

Please provide me with some help to fix this.

Also, my understanding is that I can train a model on the DataBricks MLFlow platform using sklearn libraries, and then I can store model artifacts "model.tar.gz" under the s3 directory for storing all multi-models. Now I can create a multi-model endpoint in SageMaker using the same s3 directory as the model path and using the above code. Once the endpoint is ready, I can do inference by providing the target model. Please let me know if my understanding is correct and share any relevant documents to follow for my use case.

asked 2 years ago658 views
1 Answer

The above notebook shows how to seed a pre-existing model in an already built container. This functionality could be replicated with other Amazon SageMaker Algorithms, as well as the TensorFlow and MXNet containers. Although this is certainly an easy method to bring your own model, it is not likely to provide the flexibility of a bringing your own scoring container. Please refer to other example notebooks which show how to dockerize your own training and scoring container which could be modified appropriately to your use case.

In general it is recommended to Bring your Own docker container along with your custom model, SageMaker Inference Toolkit is a library that bootstraps MMS in a way that is compatible with SageMaker multi-model endpoints, while still allowing you to tweak important performance parameters, such as the number of workers per model. The inference container in this example uses the Inference Toolkit to start MMS which can be seen in the container/ file.

In order to deep-dive further I would request you to open a support ticket with the aws premium support for further investigation in to the cloudwatch logs and the specific resource of the endpoint.

If you still have difficulties, I recommend to cut a support case and provide more detail about your account information and cloudwatch logs. Due to security reason, we cannot discuss account specific issue in the public posts.

Thank you

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions