- Newest
- Most votes
- Most comments
The mechanism for doing this with the lowest overhead for yourself would be to make use of the AWS WAF Bot Control Group. This has categories for different types of bots, such as search engines which you could use to allow that type of traffic before blocking all other traffic from India. Check the "CategorySearchEngine" section in the docs here: https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html This would remove the need for you to maintain custom rules based on user agents etc and also add further assurance using mechanisms such as reverse IP lookups to validate "good" bots are who they say they are. But note that the Bot Control groups come at an additional cost above normal WAF rules. To block traffic from a specific geolocation use the following rule type: https://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-statement-type-geo-match.html Before you do either of those however, I would validate how much of a concern blocking crawler traffic sourced from India is in reality. For example, if you look at Google's documentation, their IP ranges mainly seem to be US and Indonesia based: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot . And if you did block crawlers from a particular region, would crawlers from other countries simply take their place?
Relevant content
asked 4 years ago
asked 3 years ago
asked 3 years ago
- AWS OFFICIALUpdated a month ago
