Problem using fileds with "@" with CloudWatch Metric Filters

Question

I have been worked with CloudWatch Alarms associated with Metric Filters and so far I've been able to work without any problems. I'm generating some alarms for errors related to the AppSync API and some Glue Jobs. I haven't had any problems with the AppSync alarms, but I have with the Glue alarms, specifically, using the Log Group "/aws-glue/jobs/logs-v2", right now I've just confirmed that the Glue alarm is not being triggered because the field that I require to use in the filter patterns contains an "@", in this particular case the field is "@message", but the same thing happens with the rest of the fields in that Log Group, such as: "@log" because all of them contain a "@". If I use the metric without any pattern (only with the log group), it works propertly, likewise, with the AppSync metric it works fine, even using pattern filters because the fields do not have "@". Do you know if anyone else has had this problem and how it was solved?

re:Post Agent · Answer

The issue you're experiencing with CloudWatch Metric Filters for Glue job logs is related to the special character "@" in field names. This is a known consideration when working with CloudWatch Logs, particularly for log groups that use fields starting with "@" such as "@message" or "@log".

To resolve this issue, you need to use a specific syntax when referencing these fields in your metric filter patterns. For fields that contain non-alphanumeric characters other than the "@" symbol or the period (.), you must surround the field name with backtick characters (`).

In your case, since you're working with the "/aws-glue/jobs/logs-v2" log group and trying to use fields like "@message", you should enclose these field names in backticks in your filter pattern. For example:

Instead of using: { @message = * }
You should use: { `@message` = * }

This syntax tells CloudWatch Logs to treat the entire string within the backticks as a single field name, including the "@" symbol.

The same approach should be applied to other fields in that log group that start with "@", such as "@log". Always enclose them in backticks when defining your metric filter patterns.

By using this syntax, you should be able to create metric filters that correctly process the Glue job logs and trigger your alarms as expected. This approach allows you to work with these special field names while maintaining the functionality of your CloudWatch alarms and metric filters.

Remember to test your updated filter patterns to ensure they're matching the log events as intended. You can use the CloudWatch console to test the pattern against example log events to verify that it's working correctly before applying it to your metric filter and alarm configuration.
      
**Sources**  
[Fields that contain special characters - Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-Guidelines.html)  
[Filter pattern syntax for metric filters - Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntaxForMetricFilters.html)

Riku_Kobayashi · Answer

Hello.

I think the following logs are recorded for Glue Job, but what is the reason for including "@message" etc. in the filter?  
I thought that if you want to filter by error, just set the filter pattern to "ERROR".  
```
24/11/14 22:05:26 INFO Executor: Starting executor ID 8 on host yyy.yyy.yyy.yyy
```

Problem using fileds with "@" with CloudWatch Metric Filters

Add your answer

Relevant content