Cloudflare Lets Verified Search Crawlers Index Sites While Signaling No AI Training

TL;DR

On July 1, Cloudflare said that starting September 15, sites that block AI training would also block Googlebot, Applebot, and Bingbot, because each of those crawlers feeds search and AI from the same crawl. TechCrunch called it a deadline for the industry to split its crawlers. Deadline day came and nobody split anything. Instead, Cloudflare published an "Accountable" designation, gave it to Apple, Google, and Microsoft, and added a Disallow AI Training setting that keeps all three crawling for search while a robots.txt line asks them not to train. The hard block still exists. It now takes your search traffic with it, and you have to choose it on purpose.

The July threat

Cloudflare's July 1 post sorted bot traffic into three behaviors: Search, Agent, and Training. For every new domain, Training and Agent would be blocked by default on pages that display ads, with Search allowed. The sharp part was one sentence about crawlers with more than one job: "Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training."

That was real leverage. The same post puts "more than 20% of web domains" behind Cloudflare. TechCrunch reported that the new defaults would reach new customers, new sites from existing customers, and all existing free customers, and described the move as "a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training." Search Engine Journal put it more bluntly in its headline: Cloudflare's rules can block Googlebot.

Google's reply in July was the one it had given before. It pointed to Google-Extended, a robots.txt token that opts a site out of Gemini training without touching its place in Google Search. Googlebot itself, TechCrunch noted, crawls for Search, including AI Overviews and AI Mode.

What shipped on September 15

Googlebot is still one crawler. So are Applebot and Bingbot. What changed is Cloudflare's rulebook.

The September 15 post, by Bryan Becker, adds a Training setting called Disallow AI Training. Pick it and Bot Preference Sync writes a no-training preference into your robots.txt. Accountable mixed-use crawlers stay allowed for search. Every other training crawler gets blocked at the network, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI. Those four run separate search and training bots, so Cloudflare can stop the training bot without touching search, and it counts them as Accountable too.

The other settings got sharper. Block and Block on pages with ads used to skip mixed-use crawlers entirely, because blocking them could cost search visibility. Now both apply to them. Cloudflare says Block will stop Applebot, Bingbot, and Googlebot from reaching your site, search included.

Existing settings migrate on their own. Sites that had the legacy Block AI Bots toggle on move to Disallow AI Training for Training and Block on pages with ads for Agents. So the customers the July post said would block Googlebot now keep it, unless they go back and pick Block.

New domains get presets based on whether they monetize with ads. Sites without ads start with everything allowed. Ad-supported sites get Search allowed, Training set to Disallow AI Training, and Agent set to Block on pages with ads. The September post describes those presets and the migration of existing settings. It does not repeat the change to defaults for all existing free customers that TechCrunch reported in July.

The four-part test

To be Accountable, a crawler operator has to meet, or commit to meeting, four requirements:

• A way for site owners to opt out of AI training, through robots.txt or a similar standard.
• A way to opt out of AI summaries, set with the operator directly today and through Cloudflare next year.
• URL-level visibility into which pages were made available for training, plus metrics on how content appeared in search.
• Assurance that opting out of training will not hurt traditional search results.

The words "or commit to" matter here. Cloudflare says each of the three companies "combines capabilities available today with time-bound commitments for those still in development." Here is the split.

Google has Google-Extended in robots.txt and a toggle in Search Console that removes a site from generative AI Search features. Google started testing it on June 3, while engaging with the UK's Competition and Markets Authority, and says it reached all websites worldwide as of August 31. Opting out has a price. Google says opted-out sites "will not receive traffic or impressions from our generative AI features," which it counts at over 2.5 billion monthly users for AI Overviews and more than one billion for AI Mode. It also says the toggle is not a ranking signal for regular results. Cloudflare says Google expects URL-level transparency tools for Google-Extended "in the weeks to come."

Apple honors a robots.txt Disallow for Applebot-Extended to keep content out of foundation model training, and a nosnippet meta tag to keep it out of the AI-generated "broad world knowledge" answers in Siri and Search. Per Cloudflare, Apple has no URL-level inspection tool yet and shared details of an in-progress one for next year.

Microsoft is furthest behind on the robots.txt piece. Bing's no-training signal is still the NOARCHIVE meta tag it announced in September 2023, set page by page. Domain-level robots.txt support is targeted for early 2027. Until then, Cloudflare concedes, selecting Disallow AI Training "will not automatically convey a no-training preference to Bing through robots.txt."

The enforcement moved back into a text file

Cloudflare's own post makes the best case against its compromise. Explaining why it built network controls at all, it says of robots.txt: "Anyone can publish one, but it cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it." The network answer, it says, is to publish the preference, identify and classify the crawler, block the ones that ignore it, and report what each operator does on Cloudflare Radar.

That works when training has its own user agent, as it does for OpenAI or Anthropic. A mixed-use crawler is different. Google's documentation says Google-Extended "doesn't have a separate HTTP request user agent string"; crawling happens with Google's existing user agents, and the token is used "in a control capacity." A page fetched for the search index and a page that ends up in Gemini training arrive as the same request. Cloudflare can see Googlebot come in. It cannot see what Google does with the page afterward.

Picture a bouncer who can check IDs but not intentions. A guest says they are only here for dinner, and the bouncer has no way to know whether anyone photocopies the recipe book in the kitchen. Accountable is the list of guests whose word Cloudflare has decided to take, with Radar tracking what each one offers.

So for a site that wants search but not training, the mechanism that keeps the three biggest mixed-use crawlers out of training is the Robots Exclusion Protocol, which Martijn Koster defined in 1994 and the IETF finally wrote up as RFC 9309 in 2022. At least it is well documented.

Why the badge beat the block

You can read September 15 as a retreat or as a negotiation that worked. Cloudflare's own numbers explain why it ended here. Fewer than 1% of Cloudflare sites block search bots. 17% turn on some mechanism to block training. A default that cut Googlebot off from ad-supported pages would have forced site owners to give up search, which almost none of them want, in order to keep the training block that a sizable minority does want.

The talks did produce something. Cloudflare says it has been talking to crawler operators directly since July and that "almost all agreed" site owners deserve control and transparency. Out of that came dated commitments: Google's URL-level tools, Apple's inspection tool, Microsoft's robots.txt support. An AI summary opt-out is now a condition of the badge, not a courtesy.

Summaries are Cloudflare's next target. Its stated goal is to let site owners control how much of their content appears in AI summaries by early next year, set once on Cloudflare instead of operator by operator. It points to the IETF's AI Preferences working group as part of how that happens.

What to check on your own site

• You never touched the AI settings. Cloudflare says your current settings carry over and there is nothing to do.
• You had Block AI Bots on. You are now on Disallow AI Training and Block on pages with ads for Agents, and Googlebot, Applebot, and Bingbot keep crawling. Bing gets no robots.txt no-training signal until its support ships. NOARCHIVE covers that gap, but Bing's 2023 post says NOARCHIVE content also drops out of its chat answers.
• You want out of AI Overviews. That is Google's Search Console toggle, not Google-Extended and not a Cloudflare setting. It costs you the traffic and impressions those features send.
• You want the big three gone entirely. Select Block. It now stops Applebot, Bingbot, and Googlebot, search included.
• You run agents or scrapers. Ad-supported new domains get Agent set to Block on pages with ads by default, and there is no softer Disallow option for agents yet. Cloudflare says it will revisit that as the ai-prefs standards mature.

Key Takeaways

• No crawler split happened. Googlebot, Applebot, and Bingbot are still mixed-use, and Cloudflare labeled them Accountable instead of blocking them.
• Disallow AI Training keeps search. It writes a no-training preference into robots.txt and network-blocks every other training crawler, including the separate ones from Amazon, Anthropic, Meta, and OpenAI.
• Block now means block. Choosing Block or Block on pages with ads stops the big three mixed-use crawlers, search included.
• The training opt-out is a promise. Google-Extended has no separate user agent, so for these crawlers the network sees the same request either way and relies on each company honoring the preference.
• Microsoft lags. Bing's robots.txt no-training support is targeted for early 2027, with the page-level NOARCHIVE tag as the stopgap.
• AI Overviews are a separate switch. Google's Search Console opt-out, live worldwide since August 31, is what removes a site from AI Overviews and AI Mode.

Sources: Cloudflare: Have it both ways (Sep 15, 2026), Cloudflare: Your site, your rules (Jul 1, 2026), Cloudflare press release (Jul 1, 2026), Cloudflare: Bot Preference Sync, TechCrunch, Search Engine Journal, Google: common crawlers and Google-Extended, Google: new controls for website owners, Apple: About Applebot, Bing Webmaster Blog: NOARCHIVE and NOCACHE, RFC 9309, IETF AI Preferences working group