Amazonbot web crawler DOS incident
Has anybody had experience with the Amazonbot web crawler? (especially with it looking like a DOS incident)
We have a client where their service (hosted on AWS) kept going down until we identified an unusual amount of traffic from an Amazonbot with the User-Agent of "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)", after blocking it at an Application Load Balancer level via User-Agent Header rules (since modifying the "robots.txt" did not work / wasn't read) the response time of the clients service dropped significantly and has remained up.
If anybody knows what generally causes Amazonbot to crawl sites, or how I can identify the cause for this specific client's site, that would be extremely useful, cheers.
AmazonBot is not a service/feature of AWS.
For any questions or concerns regarding AmazonBot, I recommend reaching out to the AmazonBot team via the Contact link at the bottom of this page: https://developer.amazon.com/support/amazonbot
Hi Chris, I followed the link from the user agent with no avail currently, I did send them an email yesterday with more details of the situation as it was happening, although they have still not replied. I guess the best thing to do now is to wait for a response and be glad that the main issue has been fixed.
Crawler generates table with more columns than expectedasked 3 months ago
Can an Glue Crawler use a S3 Lambda Access Point as a data store?asked 8 days ago
Amazonbot web crawler DOS incidentasked 4 months ago
Glue Crawler CSV file with a field containing commasAccepted Answerasked a year ago
Can Glue crawler be configured to include only the most recent partition in a table?asked a month ago
Constant ERR_SIG_TIMEOUT using the web client on Linux and Windowsasked 3 years ago
S3 event notification Glue Crawler fails with Internal Service Exceptionasked 6 months ago
AWS Glue crawler creating multiple tablesasked 5 months ago
What is the best AWS service for sending SMS via REST WEB API calls?Accepted Answerasked 6 months ago
AWS Glue crawlerasked a month ago