By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

CloudFront + WAF Bots - Where are they coming from?

0

The bulk of our bill is due to bots, for example, in the last 24 hours we had 9 non-bots visits and 50 bot visits. However, the link is not indexed anywhere in Google and to get to it you need to have specific access so it's not easy for the bots to find it. I also checked Google Analytics and I can't see any referral traffic except from the page that requires specific access to view so I can't figure out where they are coming from since the link is hidden. Google Analytics also shows only 3 visits in the last 7 days so I'm not sure why WAF is showing a lot more.

We are using WAF to restrict access to the webpage hosted on CloudFront to specific IP addresses. In CloudFront under top referrers it says that 83.81% of requests are coming from "Not specified".

Is there anything we can do to stop being charged for these bot visits or to figure out where exactly they are coming from? Is it possible to block all traffic that does not come through a specific referrer without additional charges as there is only a single page (access required) which uses use to get to the restricted webpage

Topics
Security, Identity, & Compliance
Tags
AWS WAF
Language
English
LV
asked 5 months ago234 views
1 Answer
0

Hello,

A technical detail for starting, the referrer is not the source IP for the request, same as other web analytics tools can tell, what that means is that the request does not include a URL where the user click to access or was sent by. When CloudFront under top referrers it says that 83.81% of requests are "Not specified", this means the URL was accessed directly, perhaps directly from browser or via programmatic direct call, so it is likely to have this if your users have bookmarked your link or they/you are accessing the site writing the URL directly in the browser, so this "Not specified" referrer does not imply a bot access.

Was the previous point that lead you to think you are having only 9 out of 50 real users, or you detected bot accesses via AWS WAF functionality?

If you want and your website can be restricted to specific source-ip on the request, please read this article on how to allow only specific IP.

Please tell me if this helped,

profile pictureAWS
romerogt
answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content