How can I deploy a Jumpstart foundational model like llama2 and an endpoint through cloudformation?

Question

I have created a LLM inference endpoint using sagemaker studio. Making the endpoint with the studio involves choosing the model from various jumpstart models, configuring permissions, and selecting configurations for the EC2 instance it is running on. However the whole setup takes just a couple minutes. (https://aws.amazon.com/blogs/machine-learning/learn-how-to-build-and-deploy-tool-using-llm-agents-using-aws-sagemaker-jumpstart-foundation-models/)

How can I repeat this with cloudformation? I see that a way to do this is with a python sdk (https://sagemaker.readthedocs.io/en/v2.75.1/overview.html#deploy-a-pre-trained-model-directly-to-a-sagemaker-endpoint) but I'd like to make this into a stack I can stand up or tear down.

Arjun Arippa · Answer

Hi Tim,

Here is an example of a CDK deployment of a GenAI model through SageMaker JumpStart - https://aws.amazon.com/blogs/machine-learning/deploy-generative-ai-models-from-amazon-sagemaker-jumpstart-using-the-aws-cdk/

You may want to go through this blog to get a fair idea of how you can make use of CDK (which uses CloudFormation behind the scenes) to deploy the infra and code.

Thanks,
Arjun

Rashmin_K · Answer

Hello Tim,

```
```

You can leverage the following **AWS Machine Learning Blog** which discusses the deployment deployment of different models provided by Amazon SageMaker `JumpStart` (such as **Stable Diffusion model** for image generation,  the **FLAN-T5-XL model** for [natural language understanding (NLU)](https://en.wikipedia.org/wiki/Natural-language_understanding) and text generation from [Hugging Face](https://huggingface.co/)) via AWS CDK.

```

> [+] **[Deploy generative AI models from Amazon SageMaker JumpStart using the AWS CDK](https://aws.amazon.com/blogs/machine-learning/deploy-generative-ai-models-from-amazon-sagemaker-jumpstart-using-the-aws-cdk/)**
  >
  >
  >> Here’s the CDK sample referred in the above blog - **https://github.com/aws-samples/generative-ai-sagemaker-cdk-demo**

```

Hope this helps!

How can I deploy a Jumpstart foundational model like llama2 and an endpoint through cloudformation?

Relevant content