How to increase CPU Utilization of SageMaker instance

Hi,

Did you envision the AWS of Sagemaker Serverless Inferences for your use case?

https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html

Instead of you managing an instance, Sagemaker will take care of resource provisioning for you and manage up/down autoscaling so that you get more resources during peaks and less when there are limited requests. Autoscaling will allow you to remain cost-effiicent.

It doesn't work in all use cases, but give it a try to see how it works with your custom model.

This article provides all code snippets to get up & running quicky: https://medium.com/picus-security-engineering/customized-model-serving-via-aws-sagemaker-serverless-inference-a72879948321

This video may also help: https://www.youtube.com/watch?v=xIp2305saII

Best,

Didier

To improve the performance of your SageMaker instance and address the latency issues you're experiencing, consider the following strategies:

1. Use multi-model endpoints (MMEs): MMEs allow you to host multiple models on a single endpoint, which can improve resource utilization and reduce costs. With MMEs, SageMaker can dynamically load and unload models based on traffic patterns, potentially improving your overall performance.

2. Implement advanced batching strategies: Utilize dynamic batching, which groups multiple requests together before they hit the model. This can optimize GPU resource usage and balance throughput with latency. Dynamic batching instructs the server to wait for a predefined amount of time and batch up requests, which can help reduce latency while increasing throughput.

3. Consider using heterogeneous clusters: If your ML algorithm has CPU-intensive tasks, you can use SageMaker heterogeneous clusters to distribute the workload. This approach takes some CPU tasks from GPU instances and transfers them to dedicated compute-optimized CPU instances, ensuring higher GPU utilization and more efficient training.

4. Optimize instance selection: For CPU-backed instances, a higher number of vCPUs enables more concurrent invocations. For GPU-backed instances, more instance and GPU memory allows for more models to be loaded. Ensure you have some "slack" memory available for model unloading and potential rerouting in case of instance failures.

5. Explore different serving engines: SageMaker offers specialized deep learning containers (DLCs) and large model inference (LMI) containers. These containers include high-performance libraries like FasterTransformer, TensorRT-LLM, vLLM, and Transformers NeuronX, which can significantly improve inference performance.

6. Implement efficient routing strategies: Ensure that your endpoint is set up to optimally route requests to the best-suited instance, preventing bottlenecks and improving overall performance.

7. Consider quantization techniques: If applicable to your model, explore quantization methods like GPTQ, AWQ, or SmoothQuant, which can reduce model weights and improve inference speed.

8. Set up auto-scaling: Implement endpoint-level scaling policies with either custom parameters or invocations per minute to automatically add more instances to your endpoint fleet when needed.

9. Use DJLServing as the model server: This can offer per-model scaling and enable your endpoints to be traffic pattern agnostic, potentially addressing the issues you're facing with concurrent requests and increasing latency.

By implementing these strategies, you should be able to improve the performance of your SageMaker instance, reduce latency, and better handle concurrent requests. Remember to monitor your deployment using CloudWatch metrics to continually optimize your setup.
Sources
Community | Enhancing ML Efficiency with Amazon SageMaker
Instance recommendations for multi-model endpoint deployments - Amazon SageMaker
Boosting Salesforce Einstein’s code generating model performance with Amazon SageMaker | AWS Machine Learning Blog
Boost inference performance for LLMs with new Amazon SageMaker containers | AWS Machine Learning Blog
Run ML inference on unplanned and spiky traffic using Amazon SageMaker multi-model endpoints | AWS Machine Learning Blog