How to monitor GPU usage in EC2 Inf1 instance?

Question

Hello,

I tried using [Inf1 EC2 instance](https://aws.amazon.com/ec2/instance-types/inf1/) for deploying my ML model. I need to monitor the GPU usage of the ML model. I could find the CPU usage in the aws console, but not gpu usage.

Already tried:

1. https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-gpu-monitoring-gpumon.html

This didn't work. It threw this error

``` 
(python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ python3 gpumon.py 
Traceback (most recent call last): 
  File "gpumon.py", line 146, in  
    nvmlInit() 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 1450, in nvmlInit 
    nvmlInitWithFlags(0) 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 1440, in nvmlInitWithFlags 
    _nvmlCheckReturn(ret) 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 765, in _nvmlCheckReturn 
    raise NVMLError(ret) 
pynvml.nvml.NVMLError_DriverNotLoaded: Driver Not Loaded 
```

Also, `nvidia-smi` didn't work

``` 
(python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ nvidia-smi 
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

``` 
Kindly provide some help to monitor GPU usage.

Answer

Hello,

EC2 Inf1 instances feature AWS Inferentia chips (high-performance machine learning inference chips designed and built by AWS).  In order to monitor usage on these chips, you need to use tools available in the Neuron SDK as opposed to the GPU usage tools that you have tried.  A guide to all of the available profiling, debugging, and monitoring tools available for Neuron can be found here: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-tools/index.html.

Thanks!

How to monitor GPU usage in EC2 Inf1 instance?

Relevant questions

Use Nvidia GPU in AWS Batch

Machine Learning image training on EC2 with GPU

Why is the GPU not working out of the box for Deep learning AMI EC2 instance?

Relevancy of SageMaker Model Monitor for NLP?