What's the best way to monitor NVIDIA GPU utilization on Linux (Ubuntu) during model training?

Question

Which tool is recommended for monitoring NVIDIA GPU utilization on a Linux (Ubuntu) Amazon EC2 instance? I'm currently training custom TensorFlow ML models and using the NVIDIA System Management Interface ([nvidia-smi](https://developer.nvidia.com/nvidia-system-management-interface)) to track memory usage, GPU utilization, and the temperature of my NVIDIA GPU devices.

Accepted Answer

You can also use the [Amazon SageMaker Debugger Profiling Report](https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-profiling-report.html) to capture system metrics.

The report provides information on the following:
* System usage statistics
* Framework metrics
* Rule evaluation results
* Step durations
* GPU utilization
* Batch size
* CPU bottlenecks
* I/O bottlenecks
* Workload balancing
* GPU memory

What's the best way to monitor NVIDIA GPU utilization on Linux (Ubuntu) during model training?

Relevant content