Cloudwatch metrics - GetMetricData error: Maximum number of allowed metrics exceeded. Your search may have been limited

Question

## Background:
We need to display the metrics in the time range and their corresponding dimensions in the dashboard as a table, like:
| Time | Dimension1 | Dimension2 | MetricValue |
| --- | --- | --- | --- |
| 2023-1-1 12:00:00 | aa | bb | 3 |
| 2023-1-1 12:00:00 | cc | dd | 124 |

According to the [Grafana documentation](https://grafana.com/docs/grafana/latest/datasources/aws-cloudwatch/#configure-cloudwatch-settings), the metric data is obtained by calling the AWS API `GetMetricData` method, and the returned result contains up to 500 MetricDataQuery.

### Scenario 1
At 16:40 we put in 1000 different dimension data with the same metric `Metric1` by EMF. Grafana will query the data using the following statements with a time range from 16:30 to 17:30 : 
```
# with period - 30min
REMOVE_EMPTY(SEARCH('{"Namespace1/CustomMetric1","d1","d2","d3","d4","d5","d6"} MetricName="Metric1" "d1"="xxx"', 'Average', 1800))
```
##### Returned results displayed in Grafana
A total of 500 records are shown in Grafana's table and there is a warning that says `cloudwatch GetMetricData error: Maximum number of allowed metrics exceeded. Your search may have been limited`.

We understand this result because the API returns a limited number of results and only returns 500 labels.

### Scenario 2
At 17:00 we put in 12 different dimension data with the same metric `Metric1` by EMF. We use the same query as above but with a time range from 16:45 to 17:15.
##### Returned results displayed in Grafana
No data returned and with the same warning `cloudwatch GetMetricData error: Maximum number of allowed metrics exceeded. Your search may have been limited`.

### Question
I can't understand why #Scenario 2 happend, because we only have these 12 data items and we don't exceed the API limit. It is reasonable that the full 12 data should be displayed, but now there is no data returned, and I have a warning that the metrics are limited.

I can add anything that is not clear to the above explanation.

I do not know if I have some misunderstanding of certain concepts and so on, this problem has been confusing me for a long time, I hope I can get your support and help, thanks so much!

Answer

The issue you're experiencing in Scenario 2 might be related to the way that CloudWatch handles time periods in GetMetricData queries. When you specify a time range and a period, CloudWatch returns data for that period across the entire time range you've specified. This means that if your time range is shorter than your period, you might not get any data back.

In your example, you're using a period of 30 minutes and a time range of 30 minutes (from 16:45 to 17:15). This means that CloudWatch is trying to return data for a single period across the entire time range. If there is no data available for that entire period (which could be the case if your data is not evenly distributed across the time range), then you won't get any data back.

To work around this issue, you could try reducing the period to a smaller interval (e.g., 5 minutes) or expanding the time range to cover multiple periods. Alternatively, you could try using a different query that does not rely on the period parameter (e.g., using the StartTime and EndTime parameters to specify a specific time range).

Additionally, it's worth noting that the maximum number of allowed metrics in a GetMetricData query is not just based on the number of dimensions or metric names, but also on the number of periods and time ranges specified in the query. If your query includes too many periods or time ranges, you may still hit the maximum allowed metrics limit even if you're not exceeding the limit in other areas.

Cloudwatch metrics - GetMetricData error: Maximum number of allowed metrics exceeded. Your search may have been limited

Background:

Scenario 1

Returned results displayed in Grafana

Scenario 2

Returned results displayed in Grafana

Question

Relevant content