There is no universal answer, and the posts that give you one are answering a different question than the one you have. The decision turns on a single thing: whether your content is the product you sell, or the advertisement for the product you sell. Those two businesses should read the same crawler documentation and reach opposite conclusions.
What follows is the decision, not the syntax. The token list and the exact robots.txt lines are in every AI crawler token in robots.txt.
The short answer
- If your content is the product, blocking training crawlers is rational and blocking search crawlers is a revenue decision you should model before making.
- If your content is marketing for a product, blocking anything mostly costs you and protects almost nothing.
- Blocking is a preference, not a control. Cloudflare's own docs say compliance is voluntary.
- The lowest-regret position for almost everyone: block documented training tokens, allow every documented search token.
What blocking actually protects
Less than the word "blocking" suggests.
A robots.txt rule is a request that well-behaved operators honour. Cloudflare, which ships the feature most people use to write these rules, says so on its own documentation page: "robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level" (Cloudflare, retrieved 2026-08-21).
Three specific limits follow from that, and each one narrows what you are buying:
It is forward-looking only. A rule added today governs future fetches. It does not reach into a model that finished training last year, and it does not un-publish a page from a crawl archive that already shipped.
It does not cover user-initiated fetches. OpenAI, Google, and Amazon all publish versions of the same caveat. Amazon's reads: "Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives." A person who wants your page can still hand its URL to an assistant.
It does nothing about anonymous scraping. Bytespider has no operator documentation, no published IP range, and no reverse DNS pattern, so a request claiming to be Bytespider cannot even be authenticated. Rules only bind the crawlers that identify themselves honestly and choose to comply.
If what you need is content actually withheld, the tool is authentication or an edge rule, not a text file that asks nicely.
What blocking actually costs
This part is cleanly separable, which is what makes the decision tractable.
Blocking a training token costs you nothing in visibility today. GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training, and CCBot are all documented as training controls. Disallow every one and your citations in ChatGPT, Claude, Perplexity, and Apple's search surfaces are unchanged.
Blocking a search token removes you from that product's answers, immediately and completely. OpenAI's wording for the one people hit by accident: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links" (OpenAI, retrieved 2026-08-21).
Blocking Google-Extended is the one with an asymmetric cost. It costs nothing in Google Search or AI Overviews, which Google states plainly, but it does remove you from grounding in Gemini Apps, where your page could otherwise be pulled in at prompt time.
The shape of the tradeoff, then. Training blocks are close to free. Search blocks are close to total. Exactly one operator, Meta, bundles the two into a token you cannot split.
The publisher case
If people pay for your content, or advertisers pay for the attention it earns, an AI answer that summarises your article without sending the click substitutes for your revenue rather than feeding it.
Blocking training tokens is straightforwardly rational here. You are declining to donate the raw material for a product that competes with you, at zero cost to your current traffic. There is no serious argument against it beyond inertia.
Blocking search tokens is the harder call, and it is a business model question wearing a technical costume. You would be trading citations and referral clicks for the position that summarisation without payment is not acceptable. Some publishers have taken that position deliberately. That is a legitimate answer. What is not legitimate is arriving there by accident because a snippet you copied included OAI-SearchBot.
Two things worth doing before you decide: check whether AI referrals appear in your analytics at all, and check whether the pages being summarised are the ones that monetise. If the referral number is too small to attribute cleanly, that is an argument for keeping the option open rather than closing it, because you have no baseline to measure the closure against.
The SaaS case
If your content exists to get you discovered, blocking crawlers is close to self-harm.
Your documentation, your comparison pages, and your blog are advertising. Being in a model's training data means the model has some chance of knowing your product exists when a user asks an open question with no retrieval involved. Being retrievable by a search token means you can be cited with a link. You want both. The thing a publisher is protecting is the thing you are trying to give away.
The strongest counter-argument is proprietary methodology: original research, benchmark numbers, a pricing model you would rather competitors not have summarised. That is real, and the answer to it is not a robots.txt rule. It is data-nosnippet around the specific element, or authentication in front of the specific asset. Blocking a training crawler to protect a number that is on a public page is theatre, because the page is public.
For a small software company the honest default is: allow everything documented, block nothing, and spend the time you saved on whether the pages are actually citable.
The case that is not about crawlers at all
If the content you are worried about is customer data, internal documentation, or anything that should not be public, none of this applies and robots.txt is the wrong layer. A Disallow on a URL announces the URL exists. Put it behind auth and stop thinking about crawlers.
Our own answer, which we had not actually made
Until 2026-08-15, https://seobuilder.tech/robots.txt disallowed nine tokens. None of them were in this repository. They were injected at the edge by Cloudflare's Managed robots.txt feature, while src/app/robots.ts contained eight Disallow lines for authenticated paths and nothing about AI at all. The file our customers read and the file in our codebase were different documents.
By the framework above, we were a SaaS company blocking training crawlers, which is the publisher's answer applied to a business that is not a publisher. Every post here exists to be found. Being learned by a model is closer to a win for us than a loss.
The shape was not all wrong. Every search token was allowed: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot and Meta-WebIndexer were all absent from the block list, so we stayed eligible to be cited everywhere that matters. Two rows were on the wrong side. Google-Extended cost us Gemini grounding for a training protection we do not need, and meta-externalagent carries indexing along with the training we were aiming at.
So we turned it off, and now nothing is blocked. If you are a SaaS company whose growth depends on being found, that is probably your answer too. If you are a publisher whose archive is the business, it is emphatically not.
We are leaving both in place this quarter, and saying why rather than quietly fixing it. We have no measurement of Gemini Apps or Meta AI referrals from our own analytics. Removing the rules would swap one unexamined default for another. The next post that touches this will have log data behind it or it will not make a claim.
How to decide in ten minutes
- Fetch the live file.
curl -s https://yourdomain.com/robots.txt. Not the repo copy. A CDN can rewrite it after your code runs, and ours does. - Answer one question. Does a summary of my content without a click cost me money? Yes means publisher rules. No means allow almost everything.
- Split the list. Every
Disallowon a documented search token is a decision you should be able to defend in a sentence. If you cannot, delete it. - Check what you are protecting is protectable. If the answer is a public page, robots.txt is not protecting it from anything determined.
- Write down why. The reason a rule exists is the only thing that stops the next person re-litigating it, and a comment in robots.txt is free.
Our AI crawler checker runs steps one and three against a live robots.txt. It shows the operator's own wording next to each verdict, so you can see whether a token governs training or answers before you keep the rule. The robots.txt generator writes one group per token, which stops a future "block AI bots" edit from catching a search crawler. Both are free, need no account, and are our own fetch and parse.
One disclosure since we sell in this space: SEOBuilder asks seven answer engines the same buyer questions and reports which of them cited you. Google AI Overviews, Google AI Mode, ChatGPT, Perplexity, Gemini and Copilot are read as the products themselves. Claude answers through Anthropic's API rather than claude.ai.
Once you have decided, the measurement question is separate and more useful. What gets cited in AI answers covers which pages actually appear, and how to measure AI visibility covers how to count it instead of guessing. If you are also being told to add an llms.txt, llms.txt vs robots.txt vs sitemap sorts out which of the three anyone reads.