Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

How to monitor GPU usage in EC2 Inf1 instance?

1

Hello,

I tried using Inf1 EC2 instance for deploying my ML model. I need to monitor the GPU usage of the ML model. I could find the CPU usage in the aws console, but not gpu usage.

Already tried:

  1. https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-gpu-monitoring-gpumon.html

This didn't work. It threw this error

(python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ python3 gpumon.py 
Traceback (most recent call last): 
  File "gpumon.py", line 146, in <module> 
    nvmlInit() 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 1450, in nvmlInit 
    nvmlInitWithFlags(0) 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 1440, in nvmlInitWithFlags 
    _nvmlCheckReturn(ret) 
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 765, in _nvmlCheckReturn 
    raise NVMLError(ret) 
pynvml.nvml.NVMLError_DriverNotLoaded: Driver Not Loaded

Also, nvidia-smi didn't work

(python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ nvidia-smi 
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Kindly provide some help to monitor GPU usage.

Topics
Management & GovernanceMachine Learning & AICompute
Tags
AWS InferentiaAmazon CloudWatchAmazon EC2Machine Learning & AIGPU Development
Language
English

asked 4 years ago2.5K views

1 Answer
1

Hello,

EC2 Inf1 instances feature AWS Inferentia chips (high-performance machine learning inference chips designed and built by AWS). In order to monitor usage on these chips, you need to use tools available in the Neuron SDK as opposed to the GPU usage tools that you have tried. A guide to all of the available profiling, debugging, and monitoring tools available for Neuron can be found here: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-tools/index.html.

Thanks!

AWS

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content