FREE TOOL / NO ACCOUNT / RUNS ON THE PAGE YOU GIVE IT
Robots.txt tester
Is this path crawlable by this bot. Real precedence, not a prefix check: specific groups beat the wildcard outright, the longest matching pattern wins, and Allow takes a tie.
Include the query string if you have rules that match one. Rules like /*?sort= only fire when it is there.
Disallow: / on line 7 is the longest matching rule.
Googlebot → /admin/public/notice
How that was decided
A group names Googlebot, so the User-agent: * group is ignored entirely for this crawler.
- line 7Disallow/1 chars (wins)
Precedence is not first match or last match. The rule with the longest pattern wins, measured in characters exactly as written, and Allow beats Disallow when two are the same length.
Sitemaps declared
- https://example.com/sitemap.xml
QUESTIONS ABOUT THIS TOOL
- Which rule wins when Allow and Disallow both match?
- The one with the longer pattern, counted in characters exactly as written. If they are the same length, Allow wins. So Disallow: / plus Allow: /blog/ means /blog/post is crawlable, because /blog/ is six characters against one. This is the rule in RFC 9309, and it is what a naive first-match or last-match checker gets wrong.
- Why does my User-agent: * group get ignored?
- Because only one group applies to any crawler. If any group names that crawler, the wildcard group is ignored entirely, including the rules you thought were still in force. A common bug: a broad Disallow under * plus a small Googlebot group means Googlebot obeys only the small group.
- Does a group for Googlebot also cover Googlebot-News?
- Yes, unless a Googlebot-News group exists. A crawler matches a group when its own name starts with the token on the User-agent line, so Googlebot governs every Googlebot-something bot that has no group of its own. This tester implements that, which is why testing Googlebot-News against a Googlebot group returns a real answer.
- What do * and $ mean in a path?
- * stands for any run of characters, so Disallow: /*?sort= blocks any URL with that parameter anywhere. $ at the end anchors the match to the end of the path, so Disallow: /*.pdf$ blocks /file.pdf but not /file.pdf?download=1. Both are extensions on top of the original standard and every major crawler honours them.
- Does blocking a page in robots.txt remove it from Google?
- No. robots.txt stops crawling, not indexing. A blocked URL that other pages link to can still appear in results, with no description, because Google was never allowed to read it. To keep a page out of the index you have to let it be crawled and serve a noindex.
- Does this send my robots.txt anywhere?
- The matching runs entirely in your browser, so a pasted file never leaves the page. The one optional server call is Load robots.txt, which fetches that file from the site you name.
A TOOL CHECKS ONCE. SEOBUILDER KEEPS WATCH.
We can watch this for you.
A robots.txt change ships with everything else and blocks a section nobody meant to block. We check yours on a schedule and tell you the day a path that used to be crawlable stops being one.