Cafiyn Pulse
← Startup Signals
5 min readUpdated 2026-09-17Issue 3

Cloudflare now sorts bots into Search, Training and Agent. What it means for your site and your agent.

The same change lands on two audiences. Site owners can now accidentally block search; agent builders will find more doors closed.

ByKarthik KumarCafiyn Innovations
At a glance
Effective
15 September 2026, for new domains
New categories
Search, Training and Agent
Block now covers
Mixed-use crawlers, including Googlebot, Applebot and Bingbot
Ad-supported preset
Allow search, disallow training, block agents on ad pages
Agent opt-out directive
None yet, per Cloudflare

What changed

On 15 September Cloudflare replaced its single AI bot control with separate settings for Search, Training and Agent traffic. New domains onboarding from that date are offered a preset based on whether the site earns money from ads. The ad-supported preset allows search, disallows AI training, and blocks agent-category bots on pages that show ads.

Two further changes apply more widely. The Block and Block on pages with ads settings now cover mixed-use crawlers, which Cloudflare names as including Googlebot, Applebot and Bingbot, so choosing Block stops them entirely, search included. And a new Disallow AI Training option publishes a no-training preference in robots.txt while leaving search crawling allowed. Cloudflare says existing settings carry over, and that the older Block AI Bots control will be deprecated.

For site owners: the setting that costs you search

The easy mistake is choosing Block because it sounds like the strongest protection against AI training. Because it now also covers mixed-use crawlers such as Googlebot, it can remove your site from search as well. For most content sites that is a far bigger loss than whatever the setting was meant to prevent.

If your goal is to keep search traffic while opting out of training, Disallow AI Training is the setting designed for it. Check which one you have, and check it again after any dashboard change, because the names are close enough to confuse.

For agent builders: fewer pages you can read

If your product includes a browsing or retrieval agent, the default for newly onboarded ad-supported sites is now to block agent traffic on pages that show ads. That set of blocked pages will grow as new domains join Cloudflare.

Cloudflare also notes there is not yet a well-established directive that lets a site express a preference to agents the way robots.txt does for crawlers, so there is no simple way to opt back in. Reliability planning should assume some share of page fetches will fail, and fail more over time.

  • Prefer official APIs over page fetches wherever a site offers one
  • Consider licensed data sources for the domains your product depends on most
  • Look at paid access schemes such as pay-per-crawl where a publisher offers them
  • Measure your agent's fetch failure rate by domain, so a policy change shows up as a metric rather than a support ticket

What to do

  1. 1If your site is on Cloudflare, confirm your Training setting is Disallow AI Training rather than Block, unless you intend to leave search.
  2. 2If you build agents, track fetch failures by domain and add API or licensed fallbacks for the sources you rely on.
  3. 3Revisit the setting after Cloudflare deprecates Block AI Bots.

Common questions

Does Cloudflare's Block setting now block Googlebot?

Yes. Since 15 September 2026, Block and Block on pages with ads apply to mixed-use crawlers, which Cloudflare says include Googlebot, Applebot and Bingbot. Selecting Block stops them entirely, including for search.

How do I block AI training but keep search on Cloudflare?

Use the Disallow AI Training option, which publishes a no-training preference in robots.txt and leaves search crawling allowed.

Are AI agents blocked by default on Cloudflare?

For new ad-supported domains onboarding from 15 September 2026, the offered preset blocks agent-category bots on pages that show ads. Existing sites keep their current settings in almost every case, according to Cloudflare.

Sources

Read against the primary documentation rather than secondary coverage. Where a figure comes from a provider's own docs, it is quoted as published on 2026-09-17.

Related

How AI search and answer engines find your site →Vercel vs Cloudflare Pages →

More from Signals

1.8 million Android apps were scanned for hardcoded keys. Here is how to check yours.
Finding secrets in shipped apps is now a cheap, automated pipeline. Anything compiled into an APK, an IPA or a JavaScript bundle should be treated as already published.
OpenAI's Agents API will hold your agent state. What breaks if you ever need to leave.
Two major providers moved into the agent state layer in the same week. A managed harness is a real shortcut, as long as you keep a copy of what it holds.
Meta's Muse agent can now check out at your store. What merchants should change.
Agent purchases are arriving through wallets and single-use cards, not through merchants adopting a new protocol. Many merchants will receive them without choosing to.
What Brevo's SSO incident teaches about multi-tenant SAML and hijacked sending accounts.
One incident, two lessons: why email authentication did not help, and the tenant-scoping mistake anyone shipping SSO can make.
OpenAI has published seven shutdown dates. Here is the calendar.
Seven dates between 23 October and 26 February. Every one announced months ahead, in public. The failure mode is never the notice.
Nvidia bought Hugging Face. Your open-weight fallback is now a vendor relationship.
The commitments say nothing breaks. The structural change is that "we can always self-host" is now a dependency on one vendor rather than on a neutral commons.
The Videos API goes away on 24 September and OpenAI names nothing to move to.
Every other deprecation on OpenAI's page offers a migration target. This one does not, which makes it a different kind of problem.
Whisper shuts down on 26 February. The risk is not the deadline.
Five months is a generous window. Whisper is also typically wired into a background job that fails into a queue rather than into a user's face.
The Assistants API is gone. Your app may not have told you yet.
It shut down on 26 August 2026. The replacement is not a drop-in, and the most common migration mistake changes your model's behaviour without throwing an error.
One Copilot code review costs 13 requests. Here is the rest of the maths.
Copilot now bills on usage rather than seats. The headline change is simple. The multipliers underneath it are where teams get surprised.
Row-level security is on by default now. That is not the same as being protected.
Lovable fixed the default. The gap that actually leaks data was never the default, it was the policy, and no platform can write that for you.
Your model comparison is measuring the wrong number.
Every price list is per token. Your bill is per finished piece of work. Those two numbers can rank models in opposite orders.
An agent with production log access is a new identity. Treat it like one.
Always-on agents that read production logs are arriving fast. The access-control question they raise is old and well understood, which is the good news.

Open the tool.

Ten production categories, three minutes, and a ranked list of what to fix first.

Check your own stack