LimitExceededException when creating more than 5 cloudwatch metic filters with 5 space delimited fields

Question

We run a multi tenancy environment with multiple clients logging to cloudwath, and I looking for a way to pass the client name i.e client11 into the metrics stream. Given we are streaming space delimited logs into cloudwatch log groups, I created metric filters with 2 dimensions (client and log message). This works well, however I am only limited to creating 5 metric filters when using space delimited logs with regular expressions. What is causing the 5 metric filter limit and is there any other way I can pass the client name into the metric stream? Essentially I would like to create 1 metric filter per log entry we want to alert on.

**Log Snippet**

```2024-02-13 21:15:01.57 client11 [pool-57-thread-1] DEBUG o.c.portfolioService - BasicPortfolioService:getUnrealisedPnLs ```

**Metric Filter**
```
"filterName":"Combined-Configuration","metricTransformations":[{"metricValue":"1","metricNamespace":"logs","metricName":"Restart","dimensions":{"message":"$message","client":"$client"}}],"filterPattern":"[date, time, client, level, message=%deployed module Strategy.epl%]","logGroupName":"clientlogs"}
```

Answer

Hello,

I would like to inform you that, the limit for number of Metric Filters per log group is 100 [1]. However, out of these 100 Metric Filters only 5 can contain regular expression in them [1][2].

Hence, when you attempt to create more than 5 Metric filter with regular expression in them, you can observe the error:
```
Metric filter "" could not be created.
Resource limit exceeded.
```
From the provided information in the question, I could observe that you are trying to create a space delimited metric filter patter with regular expression in it as :
```
Metric Filter: [date, time, client, level, message=%deployed module Strategy.epl%]
```
And the Logged data is of format:
```
2024-02-13 21:15:01.57 client11 [pool-57-thread-1] DEBUG o.c.portfolioService - BasicPortfolioService:getUnrealisedPnLs
```
From the Metric Filter pattern provide I could observe that, the pattern is using a regex matching, however the data provided in the regex is a plain text(string).

If the message segment in the logged data will be exactly as mentioned in the filter. 
```
EXAMPLE: 2024-02-13 21:15:01.57 client11 [pool-57-thread-1] deployed module Strategy.epl 
```
Then you may consider modifying the Metric filter pattern as:
```
Metric Filter: [date, time, client, level, message="deployed module Strategy.epl"]
```
If the message segment in the logged data can have prefix and suffix along with the string / text that is to be matched.
```
EXAMPLE: 2024-02-13 21:15:01.57 client11 [pool-57-thread-1] Testing deployment, data deployed module Strategy.epl had failed!
```
Then you may consider modifying the Metric filter pattern as:
```
Metric Filter: [date, time, client, level, message="*deployed module Strategy.epl*"]
```

For detailed understanding of metric filter pattern syntax, please consider referring the documentation:

[+] Filter pattern syntax for metric filters, subscription filters, filter log events, and Live Tail - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html

References
-------------

[1] CloudWatch Logs quotas - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html

[2] Amazon CloudWatch Logs announces regular expression filter pattern syntax support - https://aws.amazon.com/about-aws/whats-new/2023/09/amazon-cloudwatch-logs-regular-expression-filter-pattern-syntax-support/

LimitExceededException when creating more than 5 cloudwatch metic filters with 5 space delimited fields

References

Relevanter Inhalt