No value to graph for a metric filter which should match hundreds of lines based on pattern test

Hi there, thanks for reaching out!

So, one thing I'd like to clarify is that when it comes to both Logs Insights queries, and when testing metric filter patterns in the process of creating a metric filter on a log group, the queries and the filter tests against log data are performed against log data that is already present in the log group. But after the metric filter is created, the metric filter pattern will only match against new incoming logs. Metrics won't be generated for past log data already in the log group.

You did note that the metric filter metric graph does report a series of 0's, indicating that it detects incoming log data to the log group, but none of the new log data that has been received since the creation of the metric filter has matched your filter pattern of:

[remote_addr, user, remote_user, timestamp, location="GET /espace-client/selfcare/contract/*", response_code=2*, body_bytes_sent, referrer, user_agent]

To be sure, I used the above metric filter on a test log group and sent log data to that log group based on the sample log you provided. I found that the metric filter did match the log data and successfully generated metric data values of 1 per log message. Otherwise, it did not match log data I sent that contained a non-matching location value or response code.

With regard to the discrepancy between the metric filter values and the results you get in Logs Insights, I see that your query against the log data is somewhat more broad than the metric filter pattern. The Insights query used will match any log message that contains the string "GET /espace-client/selfcare" whereas the metric filter pattern will only match log message that matches location value "GET /espace-client/selfcare/contract/*" AND has a response code of "2XX".

One thing you can test to verify that log data that has arrived AFTER the creation of your metric filter is indeed not matching the metric filter is to use an Insights query that would match the same log messages as the metric filter. An example of such an Insights query is:

fields @timestamp 
| parse @message "* * * [*] \"*\" * * \"*\" \"*\"" as remote_addr, user, remote_user, timestamp, location, response_code, body_bytes_sent, referrer, user_agent
| filter location like /GET \/espace-client\/selfcare\/contract*/ and response_code like /2*/
| sort @timestamp desc 

When running this query, set the time frame being searched to a range of time after the creation of the metric filter. If you receive no matches, it's possible that the incoming requests contain either a different location value or are receiving a response code other than 2XX.

Hi Justin,

Thanks a lot for your answer. Yes I know that the filter is not matching logs retroactively. it has been created two days ago and still no match :)

The search on log insight does not give any result. Yet, from your log insight test example I might have understood that the filter expects @message to be a flat space delimited string (?) but in my case it seems to be a json object so maybe that's the catch. Below is an example of event that is returned by a broader log insight search:

@message {"log":"172.17.82.30 - - [31/Mar/2022:15:14:44 +0200] \"GET /espace-client/selfcare/contract/<redacted> HTTP/1.1\" 200 8618 \"<redacted>\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15\"\n" <rest of messsage is redacted>

If its the case, I dont know how to target the @message.log property of the event in a custom metric filter which should parse one property (the log property of @message) of the json only. Something like below does not work though:

{$.log = [remote_addr, user, remote_user, timestamp, location="GET /espace-client/selfcare/contract/*", response_code=2*, body_bytes_sent, referrer, user_agent]}

AFAIK, this is a complete standard log event coming from EKS, we did not do anything to customize the event format.

Thanks again for helping! Vina.

Hi Justin,

Thank you. The following pattern works (I have added a "\"<space>" in your suggestion in order to make sure not to match requests which request byte size is 200 :) )

{$.log = "*GET /espace-client/selfcare/contract/*" && $.log = "*\" 200 *"}

Cheers, Vina.