MultiDataModel with different inference scripts

Question

I found a similar posting [here](https://repost.aws/questions/QUEVxelof3TmimoLt1Kd1SBA/how-to-configure-our-own-inference-py-for-two-different-py-torch-models-in-multi-data-model-to-build-single-endpoint-and-call-both-models-from-there) but I'm hoping my situation is a little simpler.

---

**Q: Is there a way to provide two separate inference scripts for each model in the multi-endpoint or does some dynamic/custom inference script need to be made to handle both?**

I have two model pipeline built using SageMaker Python SDK Scikit-learn processing/models:
* One is a clustering model to return cluster prediction and centroid distances when requesting inferencing
* Other is simply PCA, returning 3-components when requesting inferencing
---

Because of the odd formating of the data and the way that the output needs to be provided, both models are using custom inference scripts (e.g. predicting vs. transformation).

From what I can see in the examples for MultiDataModel, it only accepts a single entry_point for inference.py when passing the model information and just the model artifacts are "added" later:

```
cluster_model = SKLearnModel(
    model_data=cluster_artifact,
    role=role,
    entry_point="scripts/cluster_inference.py",
    sagemaker_session=sagemaker_session
)
pca_model = SKLearnModel(
    model_data=pca_artifact,
    role=role,
    entry_point="scripts/pca_inference.py",
    sagemaker_session=sagemaker_session
)
mme = MultiDataModel(
    name='model',
    model_data_prefix=model_data_prefix,
    model= cluster_model,
    sagemaker_session=sagemaker_session,
)
```
---

Deployed as separate endpoints, both perform inference as expected, but I cannot get to function as one endpoint.

Below is the most recent error I receive, but it is hard to understand where is failing, seeing as the serialization should be handled properly in my inference script and when invoking the endpoint:

```
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (503) from primary with message "{
  "code": 503,
  "type": "InternalServerException",
  "message": "Unsupported model output data type."}".
```

Any pointers or alternatives are appreciated!
Thanks

BenH · Accepted Answer

Hi there!

In short, our SageMaker scikit-learn container do not currently support model specific inference script.

The entry_point script that you referenced in your MultiDataModel object will be the inference script used for all the models. If you've added logging in your script, you will be able to see them in CloudWatch logs.

If you have some pre/post-processing script that needs to be performed on a specific model, you need to write them all in one universal inference.py . You can then add some extra attributes in your data when invoking the endpoint and have the same script read the extra attribute so it knows which pre/post-processing script to perform.

One important thing to note is that although you reference a model object in MultiDataModel, i.e.

```

mme = MultiDataModel(
    name='model',
    model_data_prefix=model_data_prefix,
    model= cluster_model,
    sagemaker_session=sagemaker_session,
)
```

The only information fetched from the model object is the image_uri and entry_point, these information is needed for during endpoint deployment.

All the model.tar.gz sitting in 'model_data_prefix' should not have a inference.py as it confuses the container and forces it to go back to default handler, hence why you're probably receiving ModelError.

Try something like below:

```
cluster_model = SKLearnModel(
    model_data=cluster_artifact,
    role=role,
    entry_point="scripts/cluster_inference.py",
    sagemaker_session=sagemaker_session
)
pca_model = SKLearnModel(
    model_data=pca_artifact,
    role=role,
    entry_point="scripts/pca_inference.py",
    sagemaker_session=sagemaker_session
)
mme = MultiDataModel(
    name='model',
    model_data_prefix=model_data_prefix,	#make sure the directory of this prefix is empty, i.e. no models in this location
    model= cluster_model,
    sagemaker_session=sagemaker_session,
)

list(mme.list_models()) # this should be empty

mme.add_model(model_data_source=cluster_artifact, model_data_path='cluster_artifact.tar.gz')	#make sure model artifact doesn't contain inference.py
mme.add_model(model_data_source=pca_artifact, model_data_path='pca_artifact.tar.gz') #make sure model artifact doesn't contain inference.py

list(mme.list_models()) # there should be two models listed now, if you look at the location of model_data_prefix, there should also be two model artifact

output_cluster = predictor.predict(data='', target_model='cluster_artifact.tar.gz')
print(output_cluster) #this should work since it's using the inference.py from cluster_inference.py

output_pca = predictor.predict(data='', target_model='pca_artifact.tar.gz')
print(output_pca) 	#this might fail since it's using cluster_inference.py, add this model's inference script into cluster_inference.py to make it work

```

I know this approach is not ideal since if you have a new model with new pre/post-processing script, you'd have to redeploy your endpoint for the new script to come into effect.

We actually just added support in our tensorflow container to allow model specific inference script: https://github.com/aws/deep-learning-containers/pull/2680

You can request for the same feature for our scikit container here: https://github.com/aws/sagemaker-scikit-learn-container/issues

MultiDataModel with different inference scripts

Relevant content