How do I analyze my CloudWatch metrics and logs for AWS WAF rules in Count mode?

Short description

Before you implement your AWS WAF rules with their final action settings, use the Count rule action to test them. AWS WAF evaluates the requests against rules set to Count. Then, AWS WAF reports matches in metrics, request samples, and logs.

You can also use Count to troubleshoot rules that generate false positives. False positives occur when a rule or a rule group blocks traffic that you don't expect it to block. For more information, see How do I detect false positives caused by AWS Managed Rules?

Resolution

To understand how CloudWatch formats metrics and logs for the Count rule action, see the following example scenarios:

Analyze logs for a custom rule set to Count

In the following example, the web ACL has a single custom rule set to Count: Geo-Count. Geo-Count counts requests if they originate from the United States. The web ACL default action is Allow.

Example Geo-Count rule:

{
  "Name": "Geo-Count",
  "Priority": 0,
  "Statement": {
    "GeoMatchStatement": {
      "CountryCodes": [
        "US"
      ]
    }
  },
  "Action": {
    "Count": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "Geo-Count"
  }
}

If the client IP address is located outside the United States, then the web ACL uses its default action and allows the request.

If the client IP address is located in the United States, then the following actions occur:

• AWS WAF counts the request.
• The CloudWatch metric for Geo-Count shows a data point in the CountedRequests metric.
  Note: CloudWatch reports the CountedRequests metric only if there's a non-zero value. To view the data in the CloudWatch console, go to Graphed metrics, and then update the statistic to Sum.
• AWS WAF uses the web ACL default action and allows the request.
• CloudWatch records a data point in the AllowedRequests metric for this web ACL.

For more information, see AWS WAF metrics and dimensions.

A custom rule set to Count is non-terminating in the logs. If the request matches a custom rule set to Count, then AWS WAF logs the following details:

• The nonTerminatingMatchingRules lists the non-terminating rules that match the request.
• The ruleId lists the rule that matches the request and is non-terminating.
• The action field is always COUNT.
• The ruleMatchDetails field includes detailed information about the rule that matches the request. AWS WAF populates this field only for SQL injection (SQLi) and cross-site scripting (XSS) match rule statements.

Example nonTerminatingMatchingRules details for a Geo-Count match:

"nonTerminatingMatchingRules": [{
  "ruleId": "Geo-Count",
  "action": "COUNT",
  "ruleMatchDetails": [

  ]
}]

For more information on all log fields, see Log fields for protection pack (web ACL) traffic.

Analyze logs for individual rules in an AWS Managed Rules rule group set to Count

To override the action of a rule inside a rule group, set one or more rules in the rule group to Count. The override changes the action so that AWS WAF counts only the requests that match. The override doesn't include rule actions in the rule group that you set to something other than Count.

If a web request matches the rule, AWS WAF first processes the match as a Count. Then, AWS WAF evaluates the subsequent rules in the rule group.

In the following example, the web ACL uses AWSManagedRulesKnownBadInputsRuleSet. There are no other rules in the web ACL, and its default action is Allow.

Example AWSManagedRulesKnownBadInputsRuleSet rule:

{
  "Name": "AWS-AWSManagedRulesKnownBadInputsRuleSet",
  "Priority": 0,
  "Statement": {
    "ManagedRuleGroupStatement": {
      "VendorName": "AWS",
      "Name": "AWSManagedRulesKnownBadInputsRuleSet",
      "ExcludedRules": [
        {
          "Name": "Log4JRCE"
        }
      ]
    }
  },
  "OverrideAction": {
    "None": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "AWS-AWSManagedRulesKnownBadInputsRuleSet"
  }
}

If you set the Log4JRCE rule to Count, then AWS WAF counts requests with a Log4j vulnerability. If the request doesn't match any other rules in the rule group, then the web ACL default action allows the request.

If you set the rule action to Count for rules inside the rule group, then they're logged as excluded rules in the rule group. You receive Count metrics for each excluded rule.

If a request matches the Log4JRCE rule, then the following actions occur:

• AWS WAF counts the request.
• The CloudWatch metric for the Log4JRCE rule shows a data point in the CountedRequests metric.

If no other rule inside the AWSManagedRulesKnownBadInputsRuleSet matches the request, then the following actions occur:

• The web ACL default action allows the request.
• CloudWatch records a data point in the AllowedRequests metric for the web ACL.

For excluded rules, the AWS WAF logs capture the following details:

• The excludedRules field lists the rules in the rule group that you have excluded. The action for these rules is set to Count.
• The exclusionType shows that the excluded rule has the action Count.
• The ruleId lists the Id of the excluded rule.

Example excludedRules details for a request that matches the Log4JRCE rule:

"ruleGroupList": [
  {
    "ruleGroupId": "AWS#AWSManagedRulesKnownBadInputsRuleSet",
    "terminatingRule": null,
    "nonTerminatingMatchingRules": [

    ],
    "excludedRules": [
      {
        "exclusionType": "EXCLUDED_AS_COUNT",
        "ruleId": "Log4JRCE"
      }
    ]
  }
]

If an individual rule inside a rule group is updated to Count, then the following actions occur:

• The logs for the request that match against this rule don't contain a Count action for the rule under the nonTerminatingMatchingRules field.
• The AWS WAF logs show this rule under the excludedRules field.

Analyze logs when an entire rule group's action is Count

If you override a rule group's action to count, then you set the entire rule group to Count.

In the following example, the web ACL uses AWSManagedRulesKnownBadInputsRuleSet. The web ACL contains no other rules. The web ACL default action is Allow. All the rules in AWSManagedRulesKnownBadInputsRuleSet use the default action and aren't set to Count. You turned on Override rule group action to Count.

Example AWSManagedRulesKnownBadInputsRuleSet rule with override to Count:

{
  "Name": "AWS-AWSManagedRulesKnownBadInputsRuleSet",
  "Priority": 0,
  "Statement": {
    "ManagedRuleGroupStatement": {
      "VendorName": "AWS",
      "Name": "AWSManagedRulesKnownBadInputsRuleSet"
    }
  },
  "OverrideAction": {
    "Count": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "AWS-AWSManagedRulesKnownBadInputsRuleSet"
  }
}

If a request matches a rule inside the AWSManagedRulesKnownBadInputsRuleSet rule group, then the following actions occur:

• CloudWatch shows a data point in the CountedRequests metric for the rule group name.
• AWS WAF processes the rest of the rules in the web ACL after the Count override takes place. CloudWatch matches the request against the web ACL default action and shows a data point in the AllowedRequests metric for the web ACL.
• In the AWS WAF logs, the matching request contains a Count action in the nonTerminatingMatchingRules field.
• The ruleGroupList shows the rule inside the rule group that originally blocked the request before the terminating action was overridden.

If a component in a request, such as a URI or query string, has a Log4j vulnerability, then the following actions occur:

• The Log4JRCE rule blocks the request.
• Because Block is a terminating action, AWS WAF stops evaluating the rule group and returns the terminating action result to the web ACL.
• Override rule group action to Count takes effect. AWS WAF overrides the rule group's terminating action to Count.
• AWS WAF processes the rest of the rules in the web ACL.
• The ruleGroupList and nonTerminatingMatchingRules details in the logs appear in the following format:

"ruleGroupList": [{
  "ruleGroupId": "AWS#AWSManagedRulesKnownBadInputsRuleSet",
  "terminatingRule": {
    "ruleId": "Log4JRCE",
    "action": "BLOCK",
    "ruleMatchDetails": null
  },
  "nonTerminatingMatchingRules": [

  ],
  "excludedRules": null
}],
"rateBasedRuleList": [

],
"nonTerminatingMatchingRules": [{
  "ruleId": "AWS-AWSManagedRulesKnownBadInputsRuleSet",
  "action": "COUNT",
  "ruleMatchDetails": [

  ]
}]

Query CloudWatch Logs Insights to analyze Count rule matches

After you deploy rules in Count mode, use Amazon CloudWatch Logs Insights to query your AWS WAF logs and identify potential false positives before you switch to Block.

Note: To use these queries, you must first turn on logging for your web ACL and send logs to a CloudWatch log group.

To find requests that triggered a specific rule in nonTerminatingMatchingRules, run the following query.

fields @timestamp
| filter @message like /"ruleId":"YOUR-RULE-SET-NAME","action":"COUNT"/
| sort @timestamp desc
| display @timestamp, terminatingRuleId, httpRequest.clientIp, httpRequest.country, @message
| limit 100

Note: Replace YOUR-RULE-SET-NAME with the name of your rule set.

To find all requests that triggered nonTerminatingMatchingRules, run the following query:

fields @timestamp, @message
| parse @message ',"nonTerminatingMatchingRules":[{"ruleId":"*","action":"*"' as rule, action
| filter action = "COUNT"
| display @timestamp, rule, action
| sort @timestamp desc

To find all requests that triggered multiple non-terminating rules along with labels, run the following query:

fields @timestamp
| filter @message like '"labels":['
| parse @message '"labels":[*]' as Labels
| parse @message '"nonTerminatingMatchingRules":[*]' as nonTermRules
| filter nonTermRules like /COUNT/
| display @timestamp, nonTermRules, Labels
| sort @timestamp desc
| limit 100

Review the query results to identify patterns in the requests that match your Count rules. If the matched requests represent legitimate traffic, then the rule generates false positives. Adjust the rule conditions before you switch the action to Block.

Related information

How AWS WAF handles rule and rule group actions in a web ACL

CloudWatch Logs Insights query language (Logs Insights QL)

How do I analyze AWS WAF logs in CloudWatch?