Yolo5 model deployment into SageMaker endpoint

Question

I have a trained yolo5 model which I deployed into Real-time end-point in SageMaker. I tried almost all gpus computes available but I could not get better than 2.6 sec for inference time. This is a light model and my target is to have < 1sec inference time. Could you please help me with any hints?
I transformed the model from pytorch to TF format.
Thank you!

Answer

Have you read the following two blog posts?

1. [Maximize TensorFlow performance on Amazon SageMaker endpoints for real-time inference](https://aws.amazon.com/blogs/machine-learning/maximize-tensorflow-performance-on-amazon-sagemaker-endpoints-for-real-time-inference/)
2. [Reduce computer vision inference latency using gRPC with TensorFlow serving on Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/reduce-compuer-vision-inference-latency-using-grpc-with-tensorflow-serving-on-amazon-sagemaker/)?

The first explores a few parameters that you can use to maximize the performance of a TensorFlow-based SageMaker real-time endpoint. These parameters are in essence overprovisioning serving processes and adjusting their parallel processing capabilities. As we saw in the tables, this overprovisioning and adjustment leads to better utilization of resources and higher throughput, sometimes an increase as much as 1,000%.

The second demonstrates how to reduce model serving latency for TensorFlow computer vision models on SageMaker via in-server gRPC communication, leading to gains of 75% in latency improvement in the example shown.

Yolo5 model deployment into SageMaker endpoint

관련 콘텐츠