Set the Bot Control rule that's blocking the requests to count.
Create a custom rule to match against the excluded rule's label and to block all matching requests except for the bot that you want to allow.
Validate that the bot traffic is allowed.
The Bot Control managed rule group verifies bots using the IP addresses from AWS WAF. If you have verified bots that route through a proxy or a CDN that doesn't preserve the client IP address while forwarding the requests, then you must specifically allow the bot.
Identify the Bot Control rule that's blocking the requests
Analyze the AWS WAF logs to identify the Bot Control rule that's blocking requests from the required bot.
2. Run the following Athena query to find the details of the request blocked by the Bot Control rule group:
waf_logs with your table name. The time interval
time > now() - interval '3' day can be replaced with your specified time interval.
WITH waf_data AS
(SELECT from_unixtime(waf.timestamp / 1000) as time,
waf.httprequest.clientip as clientip,
waf.httprequest.requestid as requestid,
waf.httprequest.country as country,
rulegroup.terminatingrule.ruleid as matchedRule,
labels as Labels,
f.value) AS kv
FROM waf_logs waf,
AS t(f), UNNEST(waf.rulegrouplist) AS t(rulegroup)
WHERE rulegroup.terminatingrule.ruleid IS NOT NULL
GROUP BY 1, 2, 3, 4, 5, 6, 7,8)
waf_data.kv['user-agent'] as UserAgent,
Where terminatingRuleId='AWS-AWSManagedRulesBotControlRuleSet' and time > now() - interval '3' day
ORDER BY time
3. (Optional) To further narrow down your search, add an additional filter on UserAgent using the AND operator in the Where clause. For a description of the fields in WAF logs, see Log Fields. For example, you can add the filter kv['user-agent'] like 'Postman%' to narrow your results.
4. Check the matchedRule column to identify the rule which is blocking the requests. Note: For additional information on Bot Control rules, see AWS WAF Bot Control rule group.
Set the Bot Control rule that's blocking the requests to count
Edit the Bot Control Rule group to set the rule that's blocking the requests to count. To set a rule to count, see Setting rule actions to count in a rule group. This allows the rule to apply its label to matching requests and to allow the bot that isn't blocked.
Create a custom rule to match against the excluded rule's label and to block all matching requests except for the bot that you want to allow
Add a label matching rule to your web ACL based on the rule label that is blocking the request. The label matching rule must come after the Bot Control managed rule group. For information on Bot Control managed rule group labels, see AWS WAF Bot Control rule group.
If a rule with the category label is blocking the request
Configure your custom rule to allow a specific blocked bot. Important: Replace the bot category and bot name labels in the rule configuration with the bot category and bot name labels from the Athena query results.
For all other rule labels
Create a custom rule to Create an exception for a blocked user agent. Important: Replace the bot signal label and the UserAgent value in the field SearchString in the rule configuration with the bot signal label and UserAgent value from the labels and UserAgent columns of Athena query results.
Validate that the bot traffic is allowed
Check the AWS WAF logs again to verify that the bot is now being allowed. If the bot is still blocked, repeat the preceding process to identify additional rules that are blocking the requests.