FREE TOOL / NO ACCOUNT / RUNS ON THE PAGE YOU GIVE IT
Robots.txt generator
Decide which AI crawlers can read your site, and see what each block costs you before you make it. Runs entirely in your browser.
AI crawlers
Everything is allowed until you block it. Each line says what blocking actually costs you.
- GPTBotOpenAI
Keeps your pages out of OpenAI model training. ChatGPT search uses a separate crawler, OAI-SearchBot, so this alone does not remove you from ChatGPT answers.
Operator documentation - ClaudeBot, anthropic-aiAnthropic
Keeps your pages out of Claude model training. Claude's search and user-triggered fetches use Claude-SearchBot and Claude-User, which this rule does not cover. anthropic-ai is an older agent name, still listed by habit.
Operator documentation - PerplexityBotPerplexity
This is the crawler Perplexity uses to surface and link sites in its answers, so blocking it takes you out of those citations.
Operator documentation - Google-ExtendedGoogle
Stops Google using your content to train Gemini and to ground its answers. Google states it does not affect inclusion in Google Search, and AI Overviews are governed by the snippet controls instead.
Operator documentation - CCBotCommon Crawl
Keeps you out of the Common Crawl corpus, which a lot of AI training sets and research datasets are built from.
Operator documentation - BytespiderByteDance
Blocks ByteDance crawling for its own AI products. No Google, ChatGPT or Perplexity visibility depends on it.
One per line. Applies to every crawler.
Your robots.txt
User-agent: * Disallow:
Save this as robots.txt at the root of your domain, so it answers at /robots.txt. It is a request, not a lock: well-behaved crawlers obey it, and it is not a security control.
QUESTIONS ABOUT THIS TOOL
- Does blocking Google-Extended remove me from AI Overviews?
- No. Google's documentation says Google-Extended controls whether your content is used to train Gemini models and to ground their answers, and that it does not affect inclusion in Google Search or work as a ranking signal. AI Overviews are part of Search, so they follow the snippet controls (nosnippet, max-snippet, data-nosnippet) instead.
- If I block GPTBot, do I disappear from ChatGPT?
- Not from ChatGPT search. OpenAI uses GPTBot for model training and OAI-SearchBot to surface sites in ChatGPT search answers, and the settings are independent. Blocking GPTBot keeps your pages out of training while ChatGPT search can still cite you.
- Where do I put the file?
- At the root of the domain, so it answers at https://yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored, and each subdomain needs its own.
- Is robots.txt a security control?
- No. It is a request that well-behaved crawlers honour, and the file itself is public, so listing a private path advertises it. Use authentication for anything that must stay private.
- Why does an allowed bot get no rule at all?
- Crawlers obey the most specific group that matches them and ignore the rest. Giving a bot its own group would make it skip your User-agent: * rules, so leaving it out is what keeps your general Disallow lines applying to it.
A TOOL CHECKS ONCE. SEOBUILDER KEEPS WATCH.
We can watch this for you.
A robots.txt is a decision you make once and then forget. We check yours on a schedule, tell you when it changes, and measure the part that matters: whether AI answers still name you for your category, and who they name instead.