Can we host AWS JumpStart Foundation models directly on AWS Inf1 or Inf2 Instances ?

Question

As the title says, we can host LLM's and Stable diffusion models from jumpstart directly on SageMaker Inf1 or Inf2 chips ?

> I tried doing that with Stable Diffusion 2 Model (i.e from studio notebook of AWS JumpStart Stable Diffusion, selected instance type as one of the AWS Inf chip). The endpoint got hosted as well but later failed at invoke endpoint step.

Answer

To deploy a model on Inf1 and Inf2 instances, you need to compile the model using AWS Neuron. In this documentation page you will find the updated list of Supported models for AWS Inferentia2, AWS Inferentia, and also AWS Trainium.

If you want to deploy Stable Diffusion on AWS Inferentia2, please see this blogpost for a full walkthrough.

Hope this helps.

jnavrro

answered 8 months ago

If you want to deploy Stable Diffusion on AWS Inferentia2, please see [this blogpost](https://aws.amazon.com/blogs/machine-learning/maximize-stable-diffusion-performance-and-lower-inference-costs-with-aws-inferentia2/) for a full walkthrough.

Hope this helps.

Can we host AWS JumpStart Foundation models directly on AWS Inf1 or Inf2 Instances ?

Relevant content