Reduce cold-start times of Async Sagemaker endpoints (HuggingFace TGI image)

Question

I'm using the HuggingFace TGI image to deploy Llama3 8b to a SageMaker async inference endpoint and using a step-scaling policy to scale down to zero and back up.

However, when scaling up from 0 in response to a request it often takes up to 20 minutes to create an instance, despite the endpoint already being In Service (responding to the cloudwatch alarm takes 1 or 2 minutes). This is roughly equivalent to just deploying the endpoint manually. I know this is a large image, but I was under the impression that since the image is ready, instances could be spun up quickly in the same manner as a normal EC2 instance. From the logs it seems to be deploying the image and re-downloading the model every time.

- Am I missing some setting/ flag that persists the image/ model?
- What advantage does async scale-to-zero offer over just deploying a real-time endpoint when a request comes in?
- Is it possible to deploy this image in ECS myself?

Giovanni Lauria · Accepted Answer

* The cold-start time you’re experiencing with your SageMaker endpoint is likely due to the time it takes to download and load the model when an instance is spun up. There isn’t a setting that persists the image/model in SageMaker.
* Asynchronous inference in SageMaker allows you to queue incoming requests and process them asynchronously. It’s useful for workloads with large payload sizes, long processing times, and near real-time latency requirements. It’s also cost-efficient as you can scale down your instances to zero when there are no requests to process.
* While it’s possible to deploy a Docker image in Amazon ECS, the process would be quite different from deploying it in SageMaker. You would need to handle aspects like load balancing, auto-scaling, and secure model data storage yourself.
* To reduce the time it takes for your endpoint to scale up in response to a request, you might want to look into defining a scaling policy that scales up from zero for new requests. However, the scaling up time might still be significant due to the reasons mentioned above.

Reduce cold-start times of Async Sagemaker endpoints (HuggingFace TGI image)

Add your answer

Relevant content