FREE TOOL / NO ACCOUNT / RUNS ON THE PAGE YOU GIVE IT
Which AI crawlers can read your page?
Paste a URL. We fetch your robots.txt, apply real group precedence to that exact path for eleven AI crawler tokens, and tell you for each one whether it is about training, about answers, or both. We also check whether the page needs JavaScript, because a crawler that does not run it sees nothing.
Free, no account, no card. We read your robots.txt and apply RFC 9309 group precedence to the exact path you gave us.
Two things almost everyone gets wrong
“Blocking Google-Extended takes you out of AI Overviews”
It does not. Google states in its crawler documentation that Google-Extended "does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search". The token governs Gemini training and grounding in Gemini Apps and Vertex AI. AI Overviews are a Search feature built on the Search index that Googlebot fills, and Google does not list them among the products this token controls. The lever for AI Overviews is the snippet controls: nosnippet, max-snippet and data-nosnippet, which also cost you ordinary Search snippets.
Read the operator’s documentation“Blocking GPTBot takes you out of ChatGPT answers”
It does not, and this is the expensive one. OpenAI states the settings are independent: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training". Search visibility hangs entirely on OAI-SearchBot, whose own line reads "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers". Blocking GPTBot alone keeps you out of training and still citable in ChatGPT, which is what most site owners actually want.
Read the operator’s documentationHow we decide whether a crawler is blocked
By the rules in RFC 9309, not by looking for the crawler’s name in the file. Group selection comes first: if robots.txt has a group naming the crawler, that group decides on its own and the User-agent: * group is ignored entirely for it. Inside the winning group the most specific rule wins, measured by the length of its path, and an Allow beats a Disallow of the same length.
That is why a file starting with User-agent: * / Disallow: / can still leave GPTBot fully allowed, and why a blanket disallow with one Allow line leaves a section crawlable. Reading the file by eye gets both of those backwards.
QUESTIONS ABOUT THIS TOOL
- Which AI crawlers does this check?
- Eleven tokens across seven operators: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google), CCBot (Common Crawl), Bytespider (ByteDance), Applebot-Extended (Apple) and meta-externalagent (Meta). For each one we say whether it is about training, about search and answers, or a fetch a person asked for, so the blocking decision is an informed one.
- Does blocking Google-Extended remove me from AI Overviews?
- No. Google-Extended controls whether your content is used for Gemini training and grounding. AI Overviews are built from Google's Search index, which Googlebot fills, and Google states Google-Extended is not used as a Search ranking signal and does not affect inclusion in Search. If you want to limit what AI Overviews can quote, the snippet controls are the lever: nosnippet, max-snippet and data-nosnippet.
- Does blocking GPTBot remove me from ChatGPT?
- No, and this is the most expensive mistake in the list. GPTBot collects training data. Search inside ChatGPT fetches pages with OAI-SearchBot, and a user who pastes your link fetches it with ChatGPT-User. Blocking GPTBot alone keeps your content out of future model training while leaving you citable in ChatGPT answers, which is what most site owners actually want.
- Why does this check a specific page and not just the site root?
- Because robots.txt rules are per path, and the answer genuinely differs. A site can allow the root and disallow /blog/, or disallow everything and carve out one section with an Allow line. We apply RFC 9309 rules to the exact path you paste: the most specific user-agent group wins over the wildcard group, and within that group the longest matching rule wins, with Allow beating Disallow on a tie.
- Why does the JavaScript check matter?
- Most AI crawlers fetch HTML and do not run JavaScript. If your content is rendered client-side, the crawler gets an empty shell no matter how permissive your robots.txt is. We count the words that actually arrive in the HTML. Under 100 words on a page carrying script tags is the signal that a text-only crawler sees close to nothing.
- Does this cost anything or need an account?
- No. It is our own HTTP request to your public robots.txt and your public page, and our own parsing of what comes back. No third-party data provider is involved, so there is nothing to bill. It is rate limited to 10 checks an hour per address.
- Should I block AI crawlers at all?
- That is a business decision, not a technical one, and the honest answer is that it depends on which token. Blocking training crawlers keeps your content out of models and costs you nothing in citations. Blocking search crawlers removes you from the answers those products give, with no training upside in exchange. This tool separates the two so you are not trading away the second while aiming at the first.
A TOOL CHECKS ONCE. SEOBUILDER KEEPS WATCH.
We can watch this for you.
robots.txt changes when someone ships, and new AI crawler tokens appear every few months. We run this check on a schedule, tell you when a crawler that could read you yesterday cannot today, and measure the part this page cannot: whether AI answers actually name you, and who they name instead.