By using AWS re:Post, you agree to the Terms of Use

How to monitor GPU usage in EC2 Inf1 instance?



I tried using Inf1 EC2 instance for deploying my ML model. I need to monitor the GPU usage of the ML model. I could find the CPU usage in the aws console, but not gpu usage.

Already tried:


This didn't work. It threw this error

(python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ python3 
Traceback (most recent call last): 
  File "", line 146, in <module> 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/", line 1450, in nvmlInit 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/", line 1440, in nvmlInitWithFlags 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/", line 765, in _nvmlCheckReturn 
    raise NVMLError(ret) 
pynvml.nvml.NVMLError_DriverNotLoaded: Driver Not Loaded 

Also, nvidia-smi didn't work

(python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ nvidia-smi 
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. 

Kindly provide some help to monitor GPU usage.

1 Answer


EC2 Inf1 instances feature AWS Inferentia chips (high-performance machine learning inference chips designed and built by AWS). In order to monitor usage on these chips, you need to use tools available in the Neuron SDK as opposed to the GPU usage tools that you have tried. A guide to all of the available profiling, debugging, and monitoring tools available for Neuron can be found here:


answered 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions