Two of our cloaking detector's first three findings were wrong

We built an internal check to answer one question: does this site serve AI crawlers something different from what it serves people? It raised three findings in its first run. Two of them were wrong, and the two wrong ones were sites doing exactly the right thing.

That result was worth more than a clean run would have been, so here is the whole of it: what we measured, what the instrument got wrong, and the one case this kind of check can never see.

Key Takeaways

• Microsoft Advertising's guide to AEO and GEO, published 6 January 2026, states the rule on page 12 of 16: the rendered DOM should contain the same facts consumers see, and you should never serve different HTML to bots.
• Across ten URLs we found zero sites serving crawlers different content. Two of the first three flags were sites refusing an unverifiable Googlebot claim, which is correct behaviour.
• Every major crawler operator publishes its source IP ranges in JSON, so a user agent string is a claim and an IP range is a check.
• Fetched twice with an identical user agent, three of eight pages returned different raw HTML. One of the three was our own pricing page.
• A checker running from outside Google's IP ranges can never see what the real Googlebot is served, so the case that matters most is the one it is blind to.
• Ten URLs we picked is not a sample of the web, and nothing here says how common cloaking is.

The rule, from the vendor that wrote it down

Cloaking is old news in search, but the AI guidance restates it for a reason: an answer engine quotes what its crawler was given, and a page can hand a crawler a different set of facts.

Microsoft Advertising's guide to AEO and GEO, published 6 January 2026, puts it on page 12 of its 16 page PDF, in a list of technical foundations: "Ensure rendered DOM contains the same facts consumers see", and "never serve different HTML to bots". We downloaded the PDF and read page 12 on 19 August 2026 rather than quoting somebody's summary of it.

The word doing the work there is facts, rather than bytes or markup, and that distinction turned out to be the difference between a check that works and one that cries wolf on every page it touches.

What we built, and what it found

The check fetches one URL several times over. Once as a browser, a second time as the same browser to establish how much the page varies on its own, once as an agent claiming nothing in particular, and once for each of five AI crawler user agents. It obeys robots.txt for each agent and skips anything disallowed. Then it renders the page in a real Chromium to see what JavaScript adds. It compares the readable text each request produced, not the HTML.

Ten URLs, a mix of our own pages, vendor documentation and well known sites:

Zero sites hid content from AI crawlers. Two varied by user agent in the harmless direction: both refuse a cookieless browser style fetch and answer the crawler agents normally, so the crawler sees more than our fake browser did. Printing that as "user agent variance" without the direction would have accused two companies of the opposite of what they do.

The two false positives

The first run raised three flags. Two were these.

The two pages were the Wikipedia article on search engine optimization and the Windows section of Microsoft support, each fetched five times with nothing changing but the user agent string:

Read the columns rather than the rows. Neither site is refusing crawlers. Both refuse something claiming to be Googlebot while serving an invented agent and a command line tool without complaint. A claimed Googlebot arriving from an address that is not Google's is what a scraper looks like, and refusing it is the defence a site is supposed to have.

We had built a check that reports correct security behaviour as an infraction, and it would have told two site owners they were cloaking.

Why that defence is correct, in one table

The reason those refusals are legitimate is that every major operator publishes the addresses its crawlers use, so anyone can verify a claim instead of trusting it. Each of these returned a JSON list of IP prefixes when we fetched it on 19 August 2026, with the list's own creation timestamp:

Google additionally documents reverse and forward DNS verification, page last updated 20 March 2026, for anyone who would rather resolve the address than match a prefix. Anthropic's crawler article, dated 7 April 2026, names its three agents and links the same kind of list.

The creation dates vary by a lot, and a list that has not moved since early 2024 is not necessarily stale, since a range that has not changed does not need republishing. What matters is that the check exists at all for every one of them.

The check that disagreed with itself

Before comparing a crawler against a browser, the tool fetches the page twice as the same browser. Whatever those two disagree about is the noise floor, and anything smaller than it is not evidence of anything.

That control is the most useful part of the whole exercise. Across eight pages fetched twice, seconds apart, with nothing changed but the passage of time:

• Raw HTML differed on three of eight pages.
• Extracted readable text differed on zero of eight.

Three pages, three unrelated causes, and we went and diffed each one:

• Our own pricing page carried a build render marker in one response and not the other, an HTML comment before the opening <html> tag. 112,938 bytes against 113,330. Our own page does not match itself.
• A Google support page returned a different CSP nonce in three places on each request. Identical byte count, identical line count, and a security feature working exactly as designed.
• Stripe's pricing page carried a different A/B experiment assignment token. Identical byte count again.

A build artefact, a security feature and an experiment framework. Not one of them altered a single word a reader sees. A checker that diffs HTML would have reported all three as differences and been wrong three times out of three, and it would have been wrong about us first.

The instrument changes what it measures

Bot defences hold state. That makes this class of check unrepeatable in a way most measurements are not.

The first run recorded support.microsoft.com refusing a plain browser fetch with a 403 while answering four of five crawler agents with a 200. Running the same check against the same URL later the same day, from the same machine, the browser was served a 200 and only the claimed Googlebot was refused. Wikipedia had also started refusing a claimed ClaudeBot by the second run, having refused only the claimed Googlebot in the first.

Nothing about those two sites changed in a few hours. The probing did. Which means a verdict from a tool like this is a statement about one moment and one requesting address, never a standing fact about a site, and it is the reason no site is named here as doing anything wrong.

The case it can never see

We can send a request that says Googlebot. We can never send one from Google's IP space. So for the single most important question, what does this site serve the real Googlebot, an external checker is blind by construction, and the sites that verify properly are exactly the ones where it is blindest.

That is why this stayed an internal script and did not become a page on this site. A public checker that cannot see the case it exists for, and that reports correct security behaviour as an infraction, is worse than no checker. The half of it that survived is the render gap: how much of a page needs JavaScript, which is observable, honest, and matters because most crawlers do not execute it. One documentation page in our run went from 22 words raw to 1,074 words rendered.

Two bugs in our own instrument, both silent

Neither of these produced an error. Both produced a clean looking result, which is the failure mode worth being afraid of.

A timeout that looked like a pass. The renderer waited for the network to go idle. One page polls continuously, so idle never arrived, the render was skipped after 20 seconds, and no render gap was reported. A page that could not be measured and a page with nothing to find printed exactly the same thing: nothing. It now waits for the DOM instead, settles for a bounded moment, and prints NOT MEASURED with the reason when a render fails.

Five lines that were one fact. Five crawler agents receiving the same 403 is one server rule, and the first version filed five separate findings for it. A report that repeats itself teaches its reader to skim, and a reader who skims misses the finding that mattered.

What to do this week

1. Verify crawlers by address, not by user agent string. Pull the JSON list for each operator above and check the source IP, or use reverse and forward DNS for Google. Blocking on the user agent alone blocks nothing, since anyone can send any string.
2. If you already refuse unverified crawler claims, keep doing it. It is correct, and any outside tool that calls it cloaking is measuring your defence rather than your content.
3. Measure your own render gap. Fetch one of your key pages with curl and count the words, then look at the same page in a browser. If the raw response is nearly empty and the rendered page is full, crawlers that do not run JavaScript are reading the empty version.
4. Check what your pages serve when nothing is claimed. An agent claiming nothing in particular is the useful control. If it is served fine and a named crawler is not, you are looking at a claim being refused, not a crawler being refused.
5. Treat any single verdict as a moment, not a fact. Run it twice, hours apart, before you believe it.

Our own check found nothing wrong with the ten sites it was pointed at, and something wrong with itself twice. The uncomfortable part is that the check was written carefully, by someone expecting false positives, and it still produced two on its first outing.

Which is the argument for measuring what an engine actually says about you rather than auditing your own pages and inferring the rest. Whether a crawler can read you is one input, and it is checkable. Whether an answer engine names you when a buyer asks is the outcome, and it is observable directly. You can run the free check on any domain with no account and no card, and see which questions name you and which name somebody else.