- Newest
- Most votes
- Most comments
AmazonBot is not a service/feature of AWS.
For any questions or concerns regarding AmazonBot, I recommend reaching out to the AmazonBot team via the Contact link at the bottom of this page: https://developer.amazon.com/support/amazonbot
I know I'm a little late but right now I'm having exactly the same problem, I checked my server log and Amazonbot is hammering my site with the same User-agent, what I found out about this issue, is one of my server sites had a bug from the CMS where everytime a visitor enter then all the links from the site will add a random string at the end for example:
mysitedomain.com/contact?string_here=SIGK91682 mysitedomain.com/company?string_here=SIGK91682 mysitedomain.com/file.pdf?string_here=SIGK91682 mysitedomain.com/file2.pdf?string_here=SIGK91682
Where "SIGK91682" will be random string and everytime it opens another link all the links inside the site will change that random strings to another one for example:
mysitedomain.com/contact?string_here=y9b3C4834 mysitedomain.com/company?string_here=y9b3C4834 mysitedomain.com/file.pdf?string_here=y9b3C4834 mysitedomain.com/file2.pdf?string_here=y9b3C4834
When Amazonbot is crawiling the site it will never end because all the links will be different every single time so Amazonbot is not detecting this problem and crawling all the pages non-stop, from my logs I can see the bot trying to scan the same url with 10000 variations now try that with all the links and files and that caused a lot of problems in the server, funny that our server is in AWS so I guess it was a win-win situation for Amazon when we had to paid them for all that data bandwidth their bot used!!
I was able to "fix" this using some modsec rules to ban the Amazonbot from using our server data bandwidth and it seems to work ok, btw Googlebot was able to no fell in the same trap I wonder why.
What I can see from our modsec log is that Amazonbot still is trying to crawl all the random links but the rules prevent it, I already trying contacting the Amazonbot team sending an email but still not an answer.
Relevant content
- asked 19 days ago
- asked 8 months ago
- asked a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated a year ago
Hi Chris, I followed the link from the user agent with no avail currently, I did send them an email yesterday with more details of the situation as it was happening, although they have still not replied. I guess the best thing to do now is to wait for a response and be glad that the main issue has been fixed.