By using AWS re:Post, you agree to the Terms of Use

No value to graph for a metric filter which should match hundreds of lines based on pattern test

1

Hi,

I find results in log insights using the following query:

fields @timestamp, @message | filter log like "GET /espace-client/selfcare" | sort @timestamp desc | limit 20

The results I find are like:

172.17.82.30 - - [29/Mar/2022:17:08:54 +0200] "GET /espace-client/selfcare/contract/<redacted> HTTP/1.1" 200 4891 "<redacted referrer>" "Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.88 Mobile Safari/537.36"

The pattern I use in my metric is:

[remote_addr, user, remote_user, timestamp, location="GET /espace-client/selfcare/contract/*", response_code=2*, body_bytes_sent, referrer, user_agent]

When I use the Test pattern feature in the metric filter creation, the above log line is matched correctly yet, when I try to create a line graph widget (metric value = 1 and default value = 0, no use of dimensions). The graph is definitely flat on a value of zero. I can tell that I have hundreds of match in log insights during one hour though. I looked for previous questions but to no avail.

What is wrong with what I am doing?

Thank you for your help.

4 Answers
1
Accepted Answer

Hi Vina,

Ah! Yeah, the space delimited log data being embedded as a JSON field would render the space delimited metric filter being used previously as unworkable. CloudWatch Logs will just see the log data in the "log" JSON field as one long string that would not be able to be parsed any further, as you found in your own test.

But luckily, we can do some simple string matching against the JSON field using wildcards! I set up and tested the following metric filter and tested revised log data based on what you've provided:

{$.log = "*GET /espace-client/selfcare/contract/*" && $.log = "* 200 *"}

This metric filter will behave similarly to the previous metric filter, in that it will look at the log field string and match only log data that contain the specific URI path as well as a "200" response. Note that a space is necessary before and after the "200" term in order to not inadvertently match against the UTC + 2 timezone the logs appear to be generated in. It is also technically possible that if a term of " 200 " appears elsewhere in the log for a request going to the "GET /espace-client/selfcare/contract/*" location, it could trigger this metric filter, but overall, I believe this should be specific enough to fit your use case.

Below are the sample logs I sent to the log group and the results:

Match

{"log":"172.17.82.30 - - [29/Mar/2022:17:08:55 +0200] \"GET /espace-client/selfcare/contract/URIPath HTTP/1.1\" 200 4891 \"http://www.google.com\" \"Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.88 Mobile Safari/537.36\""}

Did not match due to response

{"log":"172.17.82.30 - - [29/Mar/2022:17:08:56 +0200] \"GET /espace-client/selfcare/contract/URIPath HTTP/1.1\" 301 4891 \"http://www.google.com\" \"Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.88 Mobile Safari/537.36\""}

Did not match due to URI

{"log":"172.17.82.30 - - [29/Mar/2022:17:08:57 +0200] \"GET /espace-client/selfcare/URIPath HTTP/1.1\" 200 4891 \"http://www.google.com\" \"Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.88 Mobile Safari/537.36\""}

Match

{"log":"172.17.82.30 - - [29/Mar/2022:17:08:58 +0200] \"GET /espace-client/selfcare/contract/URIPath/URIPath2 HTTP/1.1\" 200 4891 \"http://www.google.com\" \"Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.88 Mobile Safari/537.36\""}

Hope this helps!

SUPPORT ENGINEER
answered 6 months ago
0

Hi there, thanks for reaching out!

So, one thing I'd like to clarify is that when it comes to both Logs Insights queries, and when testing metric filter patterns in the process of creating a metric filter on a log group, the queries and the filter tests against log data are performed against log data that is already present in the log group. But after the metric filter is created, the metric filter pattern will only match against new incoming logs. Metrics won't be generated for past log data already in the log group.

You did note that the metric filter metric graph does report a series of 0's, indicating that it detects incoming log data to the log group, but none of the new log data that has been received since the creation of the metric filter has matched your filter pattern of:

[remote_addr, user, remote_user, timestamp, location="GET /espace-client/selfcare/contract/*", response_code=2*, body_bytes_sent, referrer, user_agent]

To be sure, I used the above metric filter on a test log group and sent log data to that log group based on the sample log you provided. I found that the metric filter did match the log data and successfully generated metric data values of 1 per log message. Otherwise, it did not match log data I sent that contained a non-matching location value or response code.

With regard to the discrepancy between the metric filter values and the results you get in Logs Insights, I see that your query against the log data is somewhat more broad than the metric filter pattern. The Insights query used will match any log message that contains the string "GET /espace-client/selfcare" whereas the metric filter pattern will only match log message that matches location value "GET /espace-client/selfcare/contract/*" AND has a response code of "2XX".

One thing you can test to verify that log data that has arrived AFTER the creation of your metric filter is indeed not matching the metric filter is to use an Insights query that would match the same log messages as the metric filter. An example of such an Insights query is:

fields @timestamp 
| parse @message "* * * [*] \"*\" * * \"*\" \"*\"" as remote_addr, user, remote_user, timestamp, location, response_code, body_bytes_sent, referrer, user_agent
| filter location like /GET \/espace-client\/selfcare\/contract*/ and response_code like /2*/
| sort @timestamp desc 

When running this query, set the time frame being searched to a range of time after the creation of the metric filter. If you receive no matches, it's possible that the incoming requests contain either a different location value or are receiving a response code other than 2XX.

SUPPORT ENGINEER
answered 6 months ago
0

Hi Justin,

Thanks a lot for your answer. Yes I know that the filter is not matching logs retroactively. it has been created two days ago and still no match :)

The search on log insight does not give any result. Yet, from your log insight test example I might have understood that the filter expects @message to be a flat space delimited string (?) but in my case it seems to be a json object so maybe that's the catch. Below is an example of event that is returned by a broader log insight search:

@message {"log":"172.17.82.30 - - [31/Mar/2022:15:14:44 +0200] \"GET /espace-client/selfcare/contract/<redacted> HTTP/1.1\" 200 8618 \"<redacted>\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15\"\n" <rest of messsage is redacted>

If its the case, I dont know how to target the @message.log property of the event in a custom metric filter which should parse one property (the log property of @message) of the json only. Something like below does not work though:

{$.log = [remote_addr, user, remote_user, timestamp, location="GET /espace-client/selfcare/contract/*", response_code=2*, body_bytes_sent, referrer, user_agent]}

AFAIK, this is a complete standard log event coming from EKS, we did not do anything to customize the event format.

Thanks again for helping! Vina.

answered 6 months ago
0

Hi Justin,

Thank you. The following pattern works (I have added a "\"<space>" in your suggestion in order to make sure not to match requests which request byte size is 200 :) )

{$.log = "*GET /espace-client/selfcare/contract/*" && $.log = "*\" 200 *"}

Cheers, Vina.

answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions