AI Crawlability Checker
Enter your site URL and we fetch your robots.txt live — or paste it manually — to instantly see which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and more — are allowed or blocked from reading your site. Live fetching runs on seosorted's servers (sign in free); the paste check runs instantly in your browser.
Why AI crawlability is now a separate question from SEO
For twenty years, "is my site crawlable" meant one thing: can Googlebot read it. Now there's a second, parallel question — can the crawlers behind ChatGPT, Perplexity, Claude, and Google's own AI Overviews read it too. These are different bots with different user-agent names, and a robots.txt file can allow one while silently blocking another. A site can rank perfectly well in classic Google search while being completely invisible to every AI answer engine, simply because of a few lines in robots.txt nobody has reviewed since 2019.
This matters because if a crawler can't access your page at all, it has no way to reference or cite it in a generated answer — being crawlable doesn't guarantee citation, but being blocked guarantees you're invisible to that engine.
The AI crawlers this tool checks
- GPTBot and ChatGPT-User— OpenAI's crawlers, used for model training and live ChatGPT browsing respectively.
- ClaudeBot and anthropic-ai— Anthropic's crawlers for Claude.
- PerplexityBot— Perplexity's crawler for its answer engine.
- Google-Extended — controls whether Google can use your content for Gemini and AI Overviews training, separate from regular Googlebot search indexing.
- Applebot-Extended and CCBot (Common Crawl, used by many AI labs as a training source).
How to decide whether to allow or block these bots
There's a real tradeoff here, not an obvious right answer. Allowing AI crawlers means your content could be used to train future models (which some publishers object to) but also means it's eligible to be cited and referenced in AI-generated answers today. Blocking them protects your content from training use but makes you invisible to any AI engine that respects robots.txt. Many publishers choose a middle path: allow crawlers tied to answer-generation and citation (like PerplexityBot and ChatGPT-User) while blocking pure training crawlers.
Frequently asked questions
What is GPTBot and why would I block it?
GPTBot is OpenAI's web crawler, used to gather training data and, in some contexts, to help ChatGPT browse the web. Some site owners block it to prevent their content being used for model training; others allow it hoping to be referenced in AI-generated answers. There's a real tradeoff either way.
Does blocking AI crawlers hurt my regular Google SEO?
No. Googlebot (used for search indexing) is a separate crawler from AI-specific bots like GPTBot or Google-Extended (used for Gemini/AI training). You can block AI training crawlers while keeping full Google Search indexing, as long as you don't accidentally block Googlebot itself.
If I want my content cited by ChatGPT or Perplexity, should I allow their bots?
Generally yes — if a crawler can't access your page at all, it has no way to reference or cite it in an answer. Blocking a bot guarantees you won't show up in that engine's answers; allowing it is necessary, though not sufficient, for citation.
Is a robots.txt block absolute, or can bots ignore it?
robots.txt is a voluntary standard — well-behaved crawlers from major AI labs respect it, but it isn't a technical enforcement mechanism. A determined bad actor could ignore it entirely, though the crawlers checked by this tool are documented to comply with robots.txt rules.
Start building your content library in under 8 minutes.
Your competitors are publishing every week. Every week you don't is a week of organic traffic going to them.
No Credit Card Required.