Why isn't my AWS WAF rate-based rule blocking traffic at the threshold that I configured?

Short description

AWS WAF rate-based rules use a rolling evaluation window and check request counts approximately every 10 seconds. The rule begins to block traffic only after the next evaluation detects that the threshold count is reached. Rule priority, scope-down statements, and IP address settings can also affect the timing of the traffic blocking.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Reduce the blocking delay after requests exceed the threshold

Because AWS WAF checks request counts approximately every 10 seconds, additional requests can pass before the next evaluation window.

To reduce the number of requests that pass, take the following actions:

• To have AWS WAF detect when threshold is reached sooner, modify your rate-based rule so that the evaluation window is 60 seconds or 120 seconds.
• To handle requests that arrive during the detection period, set a lower threshold. For example, if you expect to block 200 requests, then set the threshold to 150. The valid range is 10 to 2,000,000,000.
• Create a custom AWS Lambda function that parses AWS WAF logs and adds IP addresses that exceed your threshold to an IP set rule. This approach provides near-real-time blocking because the Lambda function can respond faster than the native rate-based rule evaluation cycle. For more information about configuring logging, see Logging AWS WAF protection pack (web ACL) traffic.

Note: AWS WAF doesn't guarantee an exact limit match. For more information, see Rate-based rule caveats in AWS WAF.

Extend block durations beyond the evaluation window

Rate-based rules block an IP address only when its request count exceeds the threshold within the current window. After the count drops below the threshold, AWS WAF automatically unblocks the IP address. You can't configure a longer or permanent block duration within the rate-based rule.

To implement longer block durations, take the following actions:

• Create an IP set in AWS WAF to hold extended-block IP addresses.
• Create an IP set rule with a Block action and place it at a higher priority than your rate-based rule. A higher priority equals a lower numeric value.
• Turn on AWS WAF logging to send logs to Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), or Amazon Data Firehose.
• Create an AWS Lambda function that a CloudWatch Logs subscription filter triggers. Filter for Block actions from your rate-based rule. The function extracts the source IP address and adds it to the IP set. Store the IP address and expiration timestamp in Amazon DynamoDB.
• Create a scheduled cleanup function with Amazon EventBridge Scheduler that periodically checks Amazon DynamoDB for expired entries and removes the IP addresses from the IP set.

Verify that the rate-based rule evaluates your traffic

A higher-priority rule can apply a terminating action, such as Allow or Block. If this happens before the rate-based rule threshold is reached, then the rate-based rule doesn't evaluate the request. AWS WAF doesn't count the request toward the threshold.

To check your rule priority order, complete the following steps:

1. Open the AWS WAF console.
2. Select your web access control list (web ACL), and then choose the Rules tab.
3. Review the rule order. Rules evaluate from top to bottom. Rules with the lowest priority number evaluate first.
4. Identify Allow rules that are above your rate-based rule that match traffic you're rate limiting.

To resolve the priority conflict, take one of the following actions:

• Move the rate-based rule above Allow rules that don't exempt traffic from rate limiting.
• Change the Allow rule to Count. Count is non-terminating, so subsequent rules can evaluate the request.

Also, verify that you didn't associate the rate-based rule with multiple web ACLs. AWS WAF combines traffic from separate Application Load Balancers or Amazon CloudFront distributions that use the same rule. If you require separate thresholds, then create separate rules for each web ACL.

To check the IP addresses that the rule currently blocks, run the following get-rate-based-statement-managed-keys AWS CLI command:

aws wafv2 get-rate-based-statement-managed-keys 
    --scope REGIONAL 
    --web-acl-name YOUR-WEB-ACL-NAME 
    --web-acl-id YOUR-WEB-ACL-ID 
    --rule-name YOUR-RATE-BASED-RULE-NAME

Note: Replace YOUR-WEB-ACL-NAME with your web ACL name, YOUR-WEB-ACL-ID with your web ACL ID, and YOUR-RATE-BASED-RULE-NAME with your rate-based rule name. The preceding command returns results only for rate-based rules that aggregate on the source IP address or the X-Forwarded-For header.

Resolve users that are blocked on shared IP addresses

Multiple users can share one IP address behind a NAT gateway, proxy, or mobile carrier. If their combined requests exceed the threshold and block the shared IP address, then all users are affected.

To resolve this issue, take the following actions:

• Use the IP address in header option. If a proxy or load balancer passes the original client IP address in a header such as X-Forwarded-For, then configure the rate-based rule to use the forwarded IP address. Edit the rule and set Request aggregation to IP address in header, and then specify the header name.
• Use custom aggregation keys so that AWS WAF can't group all traffic from a shared IP address into one bucket. To create a more granular aggregation, combine the IP address with additional request attributes, such as a header value, cookie, query string, or label namespace. You can specify up to five custom keys.
• Increase the threshold to handle the aggregate traffic from the shared IP address. Review your AWS WAF logs or origin access logs to determine the estimated combined request rate from the IP address.

Determine the request count from an IP address

CloudWatch metrics for rate-based rules only show requests that AWS WAF blocked after requests exceed the threshold, not requests that passed before. To view the total traffic from a specific IP address, use AWS WAF logs.

To measure request rates, complete the following steps:

1. Turn on AWS WAF logging to send logs to CloudWatch Logs, Amazon S3, or Firehose.

2. To count requests from a specific IP address for a 5-minute window, run the following query in CloudWatch Logs Insights:

   fields @timestamp, httpRequest.clientIp, httpRequest.uri
   | filter httpRequest.clientIp = "[IP_ADDRESS]"
   | stats count(*) as requestCount by bin(5m)
   | sort @timestamp desc
   

3. To identify top talkers, run the following query:

   fields httpRequest.clientIp
   | stats count(*) as total by httpRequest.clientIp
   | sort total desc
   | limit 20
   

4. To check the IP addresses that AWS WAF currently rate limits, run the following get-rate-based-statement-managed-keys AWS CLI command:

   aws wafv2 get-rate-based-statement-managed-keys 
       --scope REGIONAL 
       --web-acl-name YOUR-WEB-ACL-NAME 
       --web-acl-id YOUR-WEB-ACL-ID 
       --rule-name YOUR-RATE-BASED-RULE-NAME
   

   Note: Replace YOUR-WEB-ACL-NAME with your web ACL name, YOUR-WEB-ACL-ID with your web ACL ID, and YOUR-RATE-BASED-RULE-NAME with your rate-based rule name. The preceding command returns results only for rate-based rules that aggregate on the source IP address or the X-Forwarded-For header.

5. Compare the output with origin access logs, such as Application Load Balancer or CloudFront logs.

Note: The sampled requests view on the AWS WAF console keeps only 3 hours of data and shows only a sample, not a complete record. To troubleshoot your issue, use the complete AWS WAF logs.