Robots.txt Validator
Paste your robots.txt, enter a URL path and a crawler name, and see exactly whether that path is allowed or blocked.
Why testing beats just reading robots.txt visually
robots.txt looks simple, but the matching logic behind it isn't always intuitive — an Allow rule can override a broader Disallow rule if it's more specific, different bots can have entirely separate rule groups, and a rule you assume applies to Googlebot might actually only apply to the wildcard group if Googlebot has no dedicated section. Reading the file top to bottom doesn't reliably tell you what a real crawler will do; simulating the actual matching logic does.
How the matching logic works
- A crawler first looks for a User-agent group that names it specifically; if none exists, it falls back to the wildcard (*) group.
- Among the rules in the matching group, the longest matching path wins — a more specific rule beats a more general one, regardless of order in the file.
- If an Allow and a Disallow rule match with equal path length, Allow generally takes precedence.
How to use this validator
Paste your live robots.txt, then test the specific URL paths you're worried about — a blog post that isn't showing up in search, an admin path you want to confirm is actually blocked, or a new AI crawler you want to check against your existing rules. Testing the exact path beats assuming from a visual read of the file.
Frequently asked questions
How does a crawler decide which rule applies when there are multiple matching rules?
The standard convention is that the longest matching path wins, regardless of whether it's an Allow or a Disallow rule. If two rules match with equal length, Allow generally takes precedence over Disallow.
What's the difference between a rule for a specific bot and the wildcard (*) group?
A crawler looks first for a User-agent group that names it specifically (like GPTBot). If none exists, it falls back to the wildcard (*) group, which applies to any crawler not otherwise named. A bot with its own specific group ignores the wildcard group entirely.
Can I test how Googlebot specifically would treat a URL?
Yes — enter "Googlebot" as the crawler name and this tool will check for a Googlebot-specific group first, falling back to the wildcard group if none exists, the same logic real crawlers use.
Why might my real crawl behavior differ from this tool's result?
This tool implements the common, widely-documented matching logic (longest match wins, specific user-agent groups override the wildcard), but some crawlers have minor implementation differences, and full wildcard pattern matching (like * and $ within paths) can vary slightly between engines.
Start building your content library in under 8 minutes.
Your competitors are publishing every week. Every week you don't is a week of organic traffic going to them.
No Credit Card Required.