Questions tagged with AWS Inferentia

As the title says, we can host LLM's and Stable diffusion models from jumpstart directly on SageMaker Inf1 or Inf2 chips ? > I tried doing that with Stable Diffusion 2 Model (i.e from studio...

We are facing issues while using this model on the aforementioned machine. We were able to run the same experiment on G5 instance successfully but we are observing that the same code is not working on...

**Can someone help me load my model to create an endpoint?** Provided explanation of steps followed, error logs and code used to create everything...thank you in advance. I'm trying very hard to...

It seems to be available according to every online source I see.

I am currently facing an issue with the AWS Neuron SDK when trying to run the PyTorch example provided in the AWS Neuron GitHub repository on a Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04)...

I am currently using Amazon SageMaker for running my machine learning models, but it is becoming costly. To reduce costs, I am considering two options: AWS Elastic Inference and AWS Inferentia. I...

Hi All, I have to compute gradient on BERT model on inferentia. For this I guess I also need access to the hidden layers. Im currently not able to proceed because of not finding literature on the net...

I'm trying to make a public facing web app that allows for inferencing, with probably ten or so available models to my users. My initial thought was that I would have a front-end basic webpage, that...

Hi, I am trying to deploy the Databricks open source LLM i.e Dolly on inf2 instance. Instance type is `inf2.24xlarge` used the AMI `Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) 2023051`. I am...

Hi, I have some code which generates a shape of torch.Size([1, 512, 1024] when calling bert on inf1. I have compiled the model for inf2. However the same code on inf2 produces a shape of...

Hi, I'm trying to run the gptj_demo on Inf2 with AMI Deep Learning AMI Neuron PyTorch 1.13.0 (Ubuntu 20.04) 20230405 and installed the pytorch neuron as...

I have an ML model from Huggingface, which essentially looks as follows: ``` import torch from transformers import BloomTokenizerFast, BloomForCausalLM device = torch.device('cuda' if...