By using AWS re:Post, you agree to the Terms of Use

How to monitor GPU usage in EC2 Inf1 instance?

1

Hello,

I tried using Inf1 EC2 instance for deploying my ML model. I need to monitor the GPU usage of the ML model. I could find the CPU usage in the aws console, but not gpu usage.

Already tried:

  1. https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-gpu-monitoring-gpumon.html

This didn't work. It threw this error

(python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ python3 gpumon.py 
Traceback (most recent call last): 
  File "gpumon.py", line 146, in <module> 
    nvmlInit() 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 1450, in nvmlInit 
    nvmlInitWithFlags(0) 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 1440, in nvmlInitWithFlags 
    _nvmlCheckReturn(ret) 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 765, in _nvmlCheckReturn 
    raise NVMLError(ret) 
pynvml.nvml.NVMLError_DriverNotLoaded: Driver Not Loaded 

Also, nvidia-smi didn't work

(python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ nvidia-smi 
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. 

Kindly provide some help to monitor GPU usage.

1 Answer
1

Hello,

EC2 Inf1 instances feature AWS Inferentia chips (high-performance machine learning inference chips designed and built by AWS). In order to monitor usage on these chips, you need to use tools available in the Neuron SDK as opposed to the GPU usage tools that you have tried. A guide to all of the available profiling, debugging, and monitoring tools available for Neuron can be found here: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-tools/index.html.

Thanks!

answered 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions