How to create a serverless endpoint in sagemaker?

Question

I am recreating an endpoint currently working in sagemaker for inference to a serverless endpoint. I am using one of the images ( huggingface-pytorch-inference:1.9.1-transformers4.12.3-cpu-py38-ubuntu20.04)  found here -> https://github.com/aws/deep-learning-containers/blob/master/available_images.md.

everything works when i choose non serverless, i.e. provisioned option for endpoint configuration , but when i try to create one with serverless option it fails.  error messages are below ( from the logs in cloudwatch) . starting with python and log4j error at the end.

'python: can't open file '/user/local/bin/deep_learning_container.py': [Errno 13] permission denied. 
Requirement already satisfied: transformers in /opt/conda/lib/pythong3.6 .....
.....
Warning: MMS is using non-default JVM parameters: -XX: -UseContainersupport
Failed to reap children process
log4j: ERROR setfile(null,true) call failed. 
java.io.FileNotFoundException: logs/mms_log.log (No Such file or directory)

why am i getting this error ???

FYI - i have set  memory to  maximum allowed memory size of 6gb. for the serverless option.

Answer

The cause might be that your SageMaker Python SDK is not updated to the latest version. Please make sure you update it to the latest version as well as the AWS SDK for Python (boto3). You can use pip:

```
pip install --upgrade boto3
pip install --upgrade sagemaker
```

For a sample notebook you can have a look [here](https://github.com/aws/amazon-sagemaker-examples/blob/main/serverless-inference/Serverless-Inference-Walkthrough.ipynb). More information on the documentation [page](https://sagemaker.readthedocs.io/en/stable/overview.html#sagemaker-serverless-inference).

How to create a serverless endpoint in sagemaker?

Relevant content