Mixtral_8x7b running slow in my server
0
I tried to run Mixtral_8x7b on AWS SageMaker 192 GB GPU and the instance name is ml.g5.48.xlagre. but it's still running slow.
132 seconds for 200 tokens.
you can see the code in the screenshot.
All the things are in place model is loaded in GPU also all the GPUs are showing that vRAM is used. But, when i start inferencing using the model only one GPU is used to process. GPU processors are not used except for the first GPU.
I have tried the following:
- torch_dtype= torch.bfloat16
- low_cpu_mem_usage=True
1回答
- 新しい順
- 投票が多い順
- コメントが多い順
0
Hi,
Do you have Instance_type = "local_gpu" ?
When you have "local", the model may default to CPU instead of GPU.
Best,
Didier
関連するコンテンツ
AWS公式更新しました 1年前
as I said earlier.
All the things are in place model is loaded in GPU. also, all the GPUs are showing that vRAM is used. But, when I start inferencing using the model only one GPU is used to process. GPU processors are not used except for the first GPU. please refer to the 2nd image("GPU-Util") I have attached in question.
Thanks Didier.