Python Script with Boto3 on CloudWatch API

Question

Hi,

I am writing a script to collect disk_used_percent metrics from CloudWatch using python boto3.
I can see all the file path disk usage on CloudWatch of a linux EC2.
But when I use Python Boto3 to call CloudWatch API, I can only see the disk_used_percent value.
All other values for the different file paths comes empty. I tried a lot but was unable to figure out why I am not gettin this value.

This is what I use to query the API.

def print_disk_usage(instance_id, path):
    metric_data = cloudwatch_client.get_metric_statistics(
        Namespace='CWAgent',
        MetricName='disk_used_percent',
        Dimensions=[
***            {'Name': 'InstanceId', 'Value': instance_id}#,
            #{'Name': 'path', 'Value': path}***
        ],
        StartTime=datetime.utcnow() - timedelta(minutes=10),
        EndTime=datetime.utcnow(),
        Period=300,
        Statistics=['Average']
    )
    datapoints = metric_data.get('Datapoints', [])
    print(datapoints)
    if datapoints:
        # Assuming we want the latest data point in the given time range
        latest_datapoint = sorted(datapoints, key=lambda x: x['Timestamp'], reverse=True)[0]
        disk_usage = latest_datapoint['Average']
        print(f"Disk usage for {path}: {disk_usage}%")
    else:
        print(f"Disk usage for {path}: No data")

If you see in the dimesion, if I comment the path, it gives me the main value for overall disk_used_percent. But I want for all file paths.
When I uncomment the commented part, it gives me empty value for all the file paths.
These values are there is CloudWatch but I cannot get it in API.
What am I doing wrong here?

Answer

To get metrics for all file paths, you need to use dimensions to filter the metrics.

Each file path would be a different dimension value. So you need to make multiple calls to get_metric_data, each time specifying a different path as the dimension value.

For example:

Make the first call with:

```
Dimensions=[
   {'Name': 'path', 'Value': 'path1'}
]
```

Make the second call with:

```
Dimensions=[  
   {'Name': 'path', 'Value': 'path2'}
]
```

And so on for each path. The API does not support returning metrics for multiple dimension values in a single call. You need to make separate calls to get the data for each path individually.

Answer

A dimension is a name/value pair that is part of the identity of a metric. The file path of the disk on the CloudWatch console for a particular EC2 instance would be one of its dimension. Hence, as it is unique to each metric and instance, the path must be specified separately in the API query for accurate results.

Reference : 
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html

In order to look into your requirement and issues you are running into, for the paths further, you can also consider reaching out to AWS via a support case.

Answer

Thank you guys for your answers. It helped me a lot.
I got result using following dimension format iterating over all the file paths, one file path at atime.

```
    'Dimensions': [
        {'Name': 'path', 'Value': '/tmp'},
        {'Name': 'InstanceId', 'Value': 'i-00xxxxxxxxxxxx'},
        {'Name': 'ImageId', 'Value': 'ami-07xxxxxxxxxxx'},
        {'Name': 'InstanceType', 'Value': 't2.micro'},
        {'Name': 'device', 'Value': 'tmpfs'},
        {'Name': 'fstype', 'Value': 'tmpfs'
    ]

```

Where the value can be changed according to what it is in CloudWatch.
You can query to find out what the dimension will be for each metrics.
These dimension format can be different for different metrics.

Python Script with Boto3 on CloudWatch API

Relevant content