Google has been explicit that crawl budget is mostly a non-issue for small to mid-sized sites (roughly under a few thousand pages), since Googlebot can typically crawl a site that size well within its normal allowance. It becomes a real, practical concern for large sites — ecommerce catalogs, large publishers, sites with heavy faceted navigation — where crawlers can waste significant budget on low-value URLs (duplicate parameter combinations, thin filtered pages, infinite calendar pages) instead of reaching genuinely important content. Managing crawl budget on a large site typically involves blocking low-value URL patterns in robots.txt, fixing redirect chains and broken links that waste crawl requests, and improving internal linking so important pages are easier to discover.
Example
A large ecommerce site discovers 60% of Googlebot's crawl requests are hitting faceted-navigation filter combinations with no unique content, leaving its actual product pages under-crawled.
Related terms
Related Reading
404 Error
A 404 error is the standard HTTP status code returned when a requested page doesn't exist on the server — the classic 'page not found' response.
A/B Testing (SEO)
A/B testing in SEO is the practice of showing two versions of a page element — usually a title tag, meta description, or on-page element — to see which one earns a better click-through rate or ranking outcome.
AI Citation
An AI citation happens when an AI assistant like ChatGPT, Perplexity, Claude, or Gemini references or links to a specific page as the source for part of its answer.
