1 réponse
- Le plus récent
- Le plus de votes
- La plupart des commentaires
1
Hi, can you please try again by changing the instance type from ml.g4dn.2xlarge to ml.g5.12xlarge. I am able to successfully load GPT-J 6B by following the steps mentioned in below article . https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-few-shot-learning.ipynb
répondu il y a 7 mois
Hopefully it should work, but the thing is purchasing a bigger instance should essentially solve the issue. But nevertheless now I have shifted to other libraries which provide very cheap and faster inference like VLLM and Llama.cpp.
Thanks for your support. Still, if there is anything in the future, this thread might help somebody, someday.
Contenus pertinents
- demandé il y a un an
- demandé il y a 2 mois
- demandé il y a un an
- demandé il y a un an
- AWS OFFICIELA mis à jour il y a 3 ans
- AWS OFFICIELA mis à jour il y a un an
Are you using SageMaker Notebooks, or Studio? I wonder if it's lack of memory, can you try with a larger instance?
Hello @Durga_S, I am using the SageMaker Notebooks. Actually the revision model "float16" already is of 12.1GB. So firstly I tried using with the 16GB memory setup. But that didn't work. Then as said in the question, I shifted to the ml.g4dn.2xlarge instance which has 32GB RAM and a T4 GPU with around 15GB RAM. But still I am unable to load the model. And I am not sure whether increasing the instance size will help out or not as I am already on the 32GB instance so this must be some another issue. Please assist, thanks.
Perhaps you can try the example here: https://github.com/marckarp/amazon-sagemaker-gptj?