Can not load GPT-J6B on 32 GB instance from HuggingFace

Question

I have used **ml.g4dn.2xlarge** instance on SageMaker to test GPT-J6B model from HuggingFace using Transformer.

I am using `revision=float16` and `low_cpu_mem_usage=True` so that the model is only of 12GB.

It is downloaded but ***after*** that it suddenly crashes the kernel.

Please share the workaround. The memory of that instance is 32 GB wit 4 vCPU.

```python
!pip install transformers

from transformers import AutoTokenizer, AutoModelForCasualLM

model = AutoModelForCasualLM.from_pretrained("EleutherAI/gpt-j-6B", revision="float16", low_cpu_mem_usage=True) # It crashes here
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")
```

It downloads 12GB model but after that, it crashes.

I tried to follow this thread [here](https://repost.aws/questions/QUsO3sfUGpTKeHiU8W9k1Kwg/why-does-my-kernal-keep-dying-when-i-try-to-import-hugging-face-bert-models-to-amazon-sage-maker) but still there I can't update the sentencepiece.

Please help.
Thanks

Answer

Hi, can you please try again by changing the instance type from ml.g4dn.2xlarge to ml.g5.12xlarge. I am able to successfully load GPT-J 6B by following the steps mentioned in below article .
https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-few-shot-learning.ipynb

Can not load GPT-J6B on 32 GB instance from HuggingFace

相關內容