GPU on g4dn.xlarge is not active / responding anymore

If your GPU on a g4dn.xlarge instance is no longer responding to Python programs using CUDA, there could be several reasons for this issue:

1. Driver Issues: The NVIDIA GPU drivers may have become corrupted or outdated. Ensure you have the latest NVIDIA Data Center drivers installed for your g4dn instance.

2. CUDA Version Mismatch: There might be a mismatch between the CUDA version installed on your system and the one your Python programs are expecting. Verify that the CUDA version in your environment matches the one your code is compiled for.

3. System Updates: Recent system updates or changes might have affected the GPU configuration. Check if any recent updates coincide with when the GPU stopped responding.

4. Resource Contention: Other processes might be monopolizing the GPU resources. Try restarting your instance to free up any locked resources.

5. Docker Container Issues: If you're using Docker, ensure that the NVIDIA Container Toolkit is properly installed and configured, and that you're using the --gpus all flag when running containers.

To troubleshoot:

1. Run nvidia-smi on the host to check if the GPU is recognized at the system level.
2. Use CUDA samples (like deviceQuery) to verify CUDA functionality.
3. Check system logs for any error messages related to the GPU or CUDA.

Regarding GPU monitoring, unlike CPU usage, GPU metrics are not natively available in the default EC2 monitoring tools. To monitor GPU usage and activity:

1. Use the nvidia-smi command directly on the instance.
2. Install and configure the NVIDIA GPU Cloud (NGC) metrics exporter.
3. Set up CloudWatch custom metrics to collect GPU data.
4. Consider third-party monitoring solutions that support GPU metrics collection.

If the issue persists after these checks, you may need to contact AWS support for further assistance, as there could be an underlying hardware problem with the GPU on your instance.
Sources
GPU Instance (g4dn.xlarge) CUDA Driver Initialization Issue in Docker Container | AWS re:Post
Only 1 GPU available in G4Dn.16xlarge EC2 instance | AWS re:Post

As per re:Post agent, you can run nvidia-smi to verify that NVIDIA driver is installed and running. If you do not get output similar to below, you will need to re-install NVIDIA driver

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       On  |   00000000:00:1E.0 Off |                    0 |
| N/A   31C    P8              9W /   70W |       1MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

where I can monitor GPU usage / activity (just like CPU)

You can install CloudWatch agent and configure GPU metrics collection. Refer to How do I send NVIDIA GPU metrics from my EC2 Linux instances to CloudWatch? for instructions.

Besides re-installing NVIDIA driver, you may want to consider replacing your EC2 instance with AWS Deep Learning AMIs which are pre-installed with NVIDIA drivers, CUDA toolkit and other libraries, and are available at no additional cost. CloudWatch agent is pre-installed, you can configured it to monitor GPU metrics.

Refer to Release Notes for available options.