Deploy LLM serverlessly on Sagemaker

Hi, from what you describe, Sagemaker Serverless Inference seems to be the right solution for your use case.

See this blog announcing the Preview: https://aws.amazon.com/blogs/machine-learning/deploying-ml-models-using-sagemaker-serverless-inference-preview/

SageMaker will automatically provision, scale, and terminate compute 
capacity based on the inference request volume. SageMaker Serverless 
Inference also means that you only pay for the duration of running the 
inference code and the amount of data processed, not for idle time. 
Moreover, you can scale to zero to optimize your inference costs.

Serverless Inference is a great choice for customers that have 
intermittent or unpredictable prediction traffic. 

It is now GA and service doc is here: https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html

Hope it helps!

Didier

Hi Xuan

Agree with Didier above, that serverless is a good option given your intermittent and low workload.

Having said that, Sagemaker serverless endpoints don't support GPUs or large instance sizes (6GB is the maximum you can request through Service Quotas). Most (if not all) LLMs would be too large, and depending on latency you may require GPU performance.

Without knowing your full requirements, I'd be looking at alternative ways to host your LLM if always-on realtime endpoints are ruled out due to cost.

Cheers James