How to get GPU metrics for each individual GPU card attached to the ec2 machine?

Question

I have an AWS account with multiple instances running. Some of the instances are g4/g5/p4 instances with multiple GPUs attached with each instance. I used the tutorial at [Link](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-gpu-monitoring-gpumon.html) to deposit the metrics for GPU utilization in AWS CloudWatch. Using the following, I could now retrieve the average utilization of all GPUs for the past hour.

```
metrics = cloudwatch_client.get_metric_statistics(
            Namespace='CWAgent',
            MetricName='nvidia_smi_utilization_gpu',
            Dimensions=[
                {'Name': 'InstanceId', 'Value': instance_id},
                # {'Name': 'index', 'Value': str(gpu_index)}
            ],
            StartTime=start_time,
            EndTime=end_time,
            Period=3600,  # 1-hour period
            Statistics=['Average']
        )
```

However, I would like to know the individual utilization of all the GPUs within the last hour. As an example, if the instance is g5.48xlarge, I want to obtain the utilization of all GPUs with index [0,1...7]. Although, I can observe this individually on the Monitoring tab under the ec2 instance, I am not able to figure out how to retrieve it through Cloudwatch SDK using boto3 as above.

![Individual GPU Monitoring](/media/postImages/original/IMEIY2hObsTRq4irn0Lpch6A)

re:Post Agent · Answer

To get GPU metrics for each individual GPU card attached to your EC2 instance, you need to modify your CloudWatch query to include the 'index' dimension. This dimension represents the individual GPU index within the instance.

Here's how you can modify your existing code to retrieve metrics for individual GPUs:

1. First, determine the number of GPUs on your instance. For a g5.48xlarge, this would be 8 GPUs (indexed 0-7).

2. Then, iterate through each GPU index and make a separate CloudWatch query for each:

```python
for gpu_index in range(8):  # Adjust the range based on the number of GPUs
    metrics = cloudwatch_client.get_metric_statistics(
        Namespace='CWAgent',
        MetricName='nvidia_smi_utilization_gpu',
        Dimensions=[
            {'Name': 'InstanceId', 'Value': instance_id},
            {'Name': 'index', 'Value': str(gpu_index)}
        ],
        StartTime=start_time,
        EndTime=end_time,
        Period=3600,  # 1-hour period
        Statistics=['Average']
    )
    # Process the metrics for each GPU here
```

By including the 'index' dimension and iterating through each GPU index, you'll be able to retrieve the individual utilization for each GPU on your instance.

Remember that to get these per-GPU metrics, you need to ensure that your CloudWatch agent is configured to collect them. In your CloudWatch agent configuration file, make sure you have the "nvidia_gpu" field under the "metrics_collected" section, and it should include the metrics you want to collect, such as "utilization_gpu".

If you're using the Deep Learning AMI, you can use the pre-configured service that collects all available GPU metrics. You can enable this by running:

```
sudo systemctl enable dlami-cloudwatch-agent@all
sudo systemctl start dlami-cloudwatch-agent@all
```

This will ensure that detailed metrics for each GPU are being collected and sent to CloudWatch, allowing you to query them as described above.  
**Sources**  
[Send NVIDIA GPU metrics from EC2 instances to CloudWatch | AWS re:Post](https://repost.aws/knowledge-center/send-ec2-nvidia-gpu-metrics-cloudwatch)  
[Monitor GPUs with CloudWatch - Deep Learning AMI](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-gpu-monitoring-gpumon.html)

How to get GPU metrics for each individual GPU card attached to the ec2 machine?

Add your answer

Relevant content