Google bots crawl rate

Question

Hello,
According to Google, we have maximized the crawl rate on our servers, which can cause excessive crawling that affects the performance of the websites.
Is there a way to limit the crawl rate? Ideally without this causing Google to eliminate products from Adwords campaigns because it cannot access the product landing pages.

khnaz35 · Answer

You can use Robots.txt

here is a sample file for you

```
User-agent: *
Disallow: /private/
Disallow: /temp/
Disallow: /test/

User-agent: Googlebot
Allow: /
Disallow: /example-subdirectory/
Crawl-delay: 10

User-agent: Bingbot
Allow: /
Disallow: /example-subdirectory/
Crawl-delay: 10

# Sitemap link
Sitemap: https://www.yoursite.com/sitemap.xml

```

Giovanni Lauria · Answer

Hi, find the official doc here on this very item. 
https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate

Google bots crawl rate

Add your answer

Relevant content