A blog post, written by Gary Illyes, has been written, called What Crawl Budget Means for Googlebot, and explains what crawl budget is, how crawl rate limits work, and what crawl demand and what factors impact a site’s crawl budget.
Gary explained that, for most sites, crawl budget is something that we wouldn’t normally have to worry about. If we’re talking about a really large site, crawl budget could become something that would have to be considered. “Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters,” Gary said.
- Crawl rate limit is designed to help Google not crawl your pages too much and too fast where it hurts your server.
- Crawl demand is how much Google wants to crawl your pages. This is based on how popular your pages are and how stale the content is in the Google index.
- Crawl budget is “taking crawl rate and crawl demand together.” Google defines “crawl budget as the number of URLs Googlebot can and wants to crawl.”
The blog post explains what factors will impact crawl budget and then some FAQs on the topic.