Crawlers work by starting from a set of known URLs (from a sitemap, previously discovered links, or a direct submission) and following every link they find outward, building a map of the web as they go. What happens after a page is crawled differs by crawler type: a traditional search crawler like Googlebot processes the page for indexing and ranking; a newer AI crawler like GPTBot or ClaudeBot fetches the page to use as training data or, in some cases, live retrieval for an AI assistant's answer. Site owners control crawler access primarily through robots.txt (which crawlers should respect, though compliance varies) and through technical factors like site speed and server response reliability, since a crawler that repeatedly times out on a slow server will crawl less of that site over time.
Example
A server monitoring tool identifies a spike in bot traffic and separates it by user-agent, showing Googlebot, Bingbot, GPTBot, and ClaudeBot all crawling the same new blog post within 48 hours of publication.
Related terms
Related Reading
404 Error
A 404 error is the standard HTTP status code returned when a requested page doesn't exist on the server — the classic 'page not found' response.
A/B Testing (SEO)
A/B testing in SEO is the practice of showing two versions of a page element — usually a title tag, meta description, or on-page element — to see which one earns a better click-through rate or ranking outcome.
AI Citation
An AI citation happens when an AI assistant like ChatGPT, Perplexity, Claude, or Gemini references or links to a specific page as the source for part of its answer.
