regex matching with MetricFilter and FilterPattern

Hi,

You should use CloudWatch Sensitive Data to achieve what you need: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html

CW defines already lots of standard expression for common PII (Personally Identifiable Info). See https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/protect-sensitive-log-data-types.html

But, you can also add your own patterns: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL-custom-data-identifiers.html

Interesting articles:

• https://aws.amazon.com/blogs/aws/protect-sensitive-data-with-amazon-cloudwatch-logs/
• https://blog.awsfundamentals.com/masking-sensitive-data-with-amazon-cloudwatch-logs-data-protection-policies

Best,

Didier

It seems you're encountering challenges with creating a valid filter pattern for your CloudWatch Logs metric filter. Let's address this issue and provide a correct approach.

When creating a metric filter pattern to match log events containing "ERROR" and a specific pattern value, you need to use the appropriate syntax that CloudWatch Logs supports. The filter pattern syntax in CloudWatch Logs is quite specific and doesn't directly support complex regular expressions.

For your case, where you want to match "ERROR" and capture a specific pattern value, you can use a filter pattern like this:

ERROR *{pattern="value"}*

Replace "value" with the actual value you're looking for. If you're trying to use a variable value from your CDK props, you would construct the filter pattern string like this:

const filterPattern = FilterPattern.literal(`ERROR *{pattern="${props.value}"}*`);

This pattern will match log events that contain "ERROR", followed by any characters, then the exact string {pattern="your_value"}, and then any additional characters.

A few important points to note:

1. The * in the pattern acts as a wildcard, matching any number of characters.
2. The curly braces and quotes in {pattern="value"} are treated as literal characters in the log event, not as special syntax.
3. Make sure that props.value doesn't contain any characters that could interfere with the pattern syntax.

If you're still encountering issues, it's possible that there might be other factors in your CDK code or CloudFormation template causing the error. In that case, you may need to review the entire resource definition and ensure all required properties are correctly set.

Remember, metric filters are only supported for log groups in the Standard log class. If you're trying to apply this to a log group in a different class, that could also cause issues.

Lastly, if you need to match more complex patterns or extract multiple values, you might need to adjust your filter pattern accordingly. CloudWatch Logs supports various filter pattern syntaxes for different types of log events, including JSON and space-delimited log events.

If you continue to face issues, consider using the TestMetricFilter operation to validate your filter pattern against sample log events before deploying it in your CDK stack.
Sources
Filter pattern syntax for metric filters, subscription filters, filter log events, and Live Tail - Amazon CloudWatch Logs
Creating metrics from log events using filters - Amazon CloudWatch Logs
MetricFilter - Amazon CloudWatch Logs