Deploying ML model on Sagemaker Endpoint for real-time inference

Question

Hello, I will really, highly appreciate your any guidance!
I have deployed an ML model on ml.g4dn.xlarge instance. The problem is that while handling >6 concurrent requests latency increases significantly.
I want to understand, are my resources fully and optimally used. According to CloudWatch logs the CPU utilization is maximum 130%, the GPU utilization is max 35%.
I do the deployment using Sagemaker module, and HuggingFace model. Here is a part of script that I use, if you need any details, please, let me know.

```
predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type='ml.g4dn.xlarge',
    endpoint_name=new_endpoint_name,
    enable_caching=True,
	serializer = wav_serializer,
	deserializer = utf8_deserializer,
)
```

Alex T · Answer

Since your overall CPU and GPU utilization are both fairly low (CPU % is measured with 100% = 1 core, if I remember correctly?), I'd say the bottleneck is likely to reside somewhere else...

It looks like you're working with uncompressed (WAV) audio, so it might be worth a quick sense check of your overall request/response volume to calculate where you're at versus your chosen [instance type's](https://aws.amazon.com/ec2/instance-types/) network bandwidth limits.

The second place I'd look would be server configuration: The default SageMaker Hugging Face inference container uses the [sagemaker-huggingface-inference-toolkit](https://github.com/aws/sagemaker-huggingface-inference-toolkit/) and the [AWSLabs' Multi-Model Server](https://github.com/awslabs/multi-model-server/blob/master/docs/configuration.md). MMS defaults various worker/thread counts based on the number of CPUs & GPUs available on the instance, and it looks like you're running a relatively small model / high number of requests (6 concurrent) versus the instance's vCPUs (4) and GPUs (1). You could try bumping these thread/worker counts up by e.g. setting environment variables `MMS_NUMBER_OF_NETTY_THREADS`, `MMS_NETTY_CLIENT_THREADS`, `MMS_DEFAULT_WORKERS_PER_MODEL` - in case the problem is some part of the server running out of threads while the processors are still idle... But at some point the CPU-GPU communication itself could become the bottleneck.

If you're still struggling to squeeze out the performance you need with the MMS configuration options (and are pretty confident the bottleneck is in the serving engine rather than maxing out some other resource on your instance like network bandwidth etc), you could explore some of the other optimized server options on SageMaker, like the [DJL](https://sagemaker.readthedocs.io/en/stable/frameworks/djl/using_djl.html)/[LMI container](https://docs.djl.ai/master/docs/serving/serving/docs/lmi/index.html), [Triton Inference Server](https://docs.aws.amazon.com/sagemaker/latest/dg/triton.html), or [Hugging Face TGI](https://aws.amazon.com/blogs/machine-learning/announcing-the-launch-of-new-hugging-face-llm-inference-containers-on-amazon-sagemaker/). These each have their own configuration options and tend to require a bit of adjustment to your model package, so there might be some effort involved to try them out... But optimally queuing and mapping web requests through (Python pre/post-processing and?) GPU-accelerated models is a hard problem multiple projects are trying to solve with different trade-offs - so it's useful to have the options available at least!

Giovanni Lauria · Answer

It seems to me that your GPU is underutilized, thus suggesting  a change of instance switching from GPU to more CPU.

Anyway here is a link on how to nvidia G4 instances.

https://developer.nvidia.com/blog/getting-the-most-out-of-nvidia-t4-on-aws-g4-instances/

Deploying ML model on Sagemaker Endpoint for real-time inference

Add your answer

Relevant content