By using AWS re:Post, you agree to the Terms of Use
/How can you filter by json property in cloudwatch log?/

How can you filter by json property in cloudwatch log?

0

I have json being logged in cloudwatch log. I am trying to filter by specific json property value. I am using the dot notation. It never finds what I know is there.

This is in the log

{ "requestId": "abc123", "apiId": "123456"}

I am filtering with this

{$.requestId="abc123"}

It never finds the log item. Is this supposed to work?

  • This log insights query returns all the message that has pb-0c25558e52262bc37. fields eventName, eventSource, errorCode, userIdentity.arn |filter @message like /(?i)(pb-0c25558e52262bc37)/ |limit 200

2 Answers
1

Try to run it from Logs Insights and using smth like filter requestId="abc123" Some doc is here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_RunSampleQuery.html

answered 7 months ago
0

Hi Andyn,

That {$.requestId="abc123"} looks like a valid filter for JSON formatted logs, and yes it is supported and it should work if the log event(s) are in the log group. You can check more details from (1).

I just checked the 'log event' and 'filter pattern' you have provided which is working as expected on my end with replication. Have you checked the retention policy of the Log Group ? and whether the log event(s) that should be matching are recent(not past the Retention policy period)?

For instance, to check whether your filter pattern is working or not you can setup a test Log Group/Log Stream and create log event using the Console. The steps are -> CW Console -> Log Groups -> go into the Log Stream -> Action -> Create log event. Then you can test with adjusting the filter with matching and non-matching patterns to see if the filter works or not.

If you are still encountering problems filtering the log events, please try with specific and short time ranges rather than just filtering with default settings on the console. Further, if you have a Premium Support subscription consider creating Technical Support case under CloudWatch Logs service so that we can assist you in a private setting.

(1) Filter and pattern syntax - Matching terms in JSON log events - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html#matching-terms-events-json

Thanks

SUPPORT ENGINEER
answered 7 months ago
  • The filter works when I manually create the log event and search it. Problem seems to be searching older events. They just dont show up. I have log events from 2 weeks ago. I set the search range to 4 weeks. Still does not find it. I am testing this because I noticed that our log group metric filter alarms are not working because the filter is not catching events. The filter works when I test them when creating the alarm, but when I test them manually in log group filter it does not work. The alarms are not getting triggered because the {$.xxx} filter thing is not working. I will probably need to resort to straight text string matching. Something definitely wrong here with filtering.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions