How can make multi model endpoint with SageMaker?

Question

This is my code.

```python
from datetime import datetime
from sagemaker.multidatamodel import MultiDataModel
mme = MultiDataModel(
    name="LV-multi-" + datetime.now().strftime("%Y-%m-%d-%H-%M-%S"),
    model_data_prefix=model_dir, # 2에서 구한 모델이 모여있는 폴더(경로)!!,
    model=sagemaker_model,  # 모델 객체 1개 우선 넣기
    sagemaker_session=sess
)

predictor = mme.deploy(
    initial_instance_count=1,
    instance_type="ml.g4dn.xlarge"
)
```

And error message.
How can I find Ecr Image(within multi-models=true)?
```
ClientError: An error occurred (ValidationException) when calling the CreateModel operation: Your Ecr Image 763104351884.dkr.ecr.ap-northeast-2.amazonaws.com/pytorch-inference:1.8.1-gpu-py3 does not contain required com.amazonaws.sagemaker.capabilities.multi-models=true Docker label(s).
```

Heiko · Accepted Answer

Hi there - thanks for opening this thread. Multi-model endpoints are not supported on GPU instance types, see here: https://docs.aws.amazon.com/sagemaker/latest/dg/multi-model-endpoints.html#multi-model-endpoint-instance

In order to host a multi-model endpoint, choose a CPU instance type instead. The ECR image for CPUs will contain the required  `com.amazonaws.sagemaker.capabilities.multi-models=true` label, see here: https://github.com/aws/deep-learning-containers/blob/master/pytorch/inference/docker/1.8/py3/Dockerfile.cpu

How can make multi model endpoint with SageMaker?

Relevant content