Mixtral_8x7b running slow in my server
0
I tried to run Mixtral_8x7b on AWS SageMaker 192 GB GPU and the instance name is ml.g5.48.xlagre. but it's still running slow.
132 seconds for 200 tokens.
you can see the code in the screenshot.
All the things are in place model is loaded in GPU also all the GPUs are showing that vRAM is used. But, when i start inferencing using the model only one GPU is used to process. GPU processors are not used except for the first GPU.
I have tried the following:
- torch_dtype= torch.bfloat16
- low_cpu_mem_usage=True
1 個回答
- 最新
- 最多得票
- 最多評論
0
Hi,
Do you have Instance_type = "local_gpu" ?
When you have "local", the model may default to CPU instead of GPU.
Best,
Didier
相關內容
AWS 官方已更新 2 年前- AWS 官方已更新 1 年前
- AWS 官方已更新 2 年前
as I said earlier.
All the things are in place model is loaded in GPU. also, all the GPUs are showing that vRAM is used. But, when I start inferencing using the model only one GPU is used to process. GPU processors are not used except for the first GPU. please refer to the 2nd image("GPU-Util") I have attached in question.
Thanks Didier.