Mixtral_8x7b running slow in my server
0
I tried to run Mixtral_8x7b on AWS SageMaker 192 GB GPU and the instance name is ml.g5.48.xlagre. but it's still running slow.
132 seconds for 200 tokens.
you can see the code in the screenshot.
All the things are in place model is loaded in GPU also all the GPUs are showing that vRAM is used. But, when i start inferencing using the model only one GPU is used to process. GPU processors are not used except for the first GPU.
I have tried the following:
- torch_dtype= torch.bfloat16
- low_cpu_mem_usage=True
1 Answer
- Newest
- Most votes
- Most comments
0
Hi,
Do you have Instance_type = "local_gpu" ?
When you have "local", the model may default to CPU instead of GPU.
Best,
Didier
Relevant content
- asked 5 months ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 10 months ago
as I said earlier.
All the things are in place model is loaded in GPU. also, all the GPUs are showing that vRAM is used. But, when I start inferencing using the model only one GPU is used to process. GPU processors are not used except for the first GPU. please refer to the 2nd image("GPU-Util") I have attached in question.
Thanks Didier.