There is a checkbox in the Cloudflare dashboard that says block AI training crawlers.

Tick it and you also remove Googlebot, Applebot and BingBot. Your site leaves the search index.

That is not a bug and it is not a trick. It is the only coherent answer Cloudflare can give when one crawler does two jobs and you allow one of them while forbidding the other. But it means a content decision and a search visibility decision now share a single control, and on September 15 that control starts moving on its own for a large slice of the web.

What actually flips on September 15

Cloudflare announced this on July 1, splitting automated traffic three ways.

Search indexes your content so it can answer questions about it later, and sends referral traffic back. Agent is automated activity acting in real time for a person, the fetch bot that pulls a page because someone asked a chatbot a question. Training takes your content to train or fine-tune a model.

From September 15, on pages that display ads, Training and Agent are blocked by default. Search stays open.

The new defaults reach new domains onboarding to Cloudflare, new customers, and every existing customer on the free tier. Paid customers with configured sites keep the rules they already set.

Matthew Prince framed it as an ecosystem problem: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.”

The collision nobody budgeted for

Cloudflare says multi-purpose crawlers, the ones combining Search with Training, are allowed or blocked according to all of their behaviors, and that defaults are enforced by the most restrictive applicable rule.

Then it names the three that matter. Googlebot. Applebot. BingBot.

So the strictest rule wins, and the strictest rule is the one you set to protect your corpus. An infrastructure engineer ticks a box on a Tuesday afternoon, marketing finds out five weeks later when impressions have already fallen off, and by then nobody remembers the change was made.

Cloudflare’s own community forum has threads from people hitting this before the deadline. The mechanism is documented. The surprise is entirely on the operator side.

Content Signals, and a different billing event

Alongside the categories, Cloudflare extended Content Signals, which already existed: three machine-readable directives in robots.txt. search for building an index, ai-input for feeding real-time answers, ai-train for training or fine-tuning. Same taxonomy, expressed in a file you own whether or not you use Cloudflare. That portability is worth more than the dashboard toggle.

Pay Per Crawl also became Pay Per Use. The old model charged bots per fetch. The new one pays publishers when their content is used inside a generated answer. Ceramic.ai and You.com are the named launch partners.

Moving the billing event from the request to the use is the right direction, and it changes what you have to measure. Fetches are countable. Citations are not, at least not the way most teams try. We wrote about that when Muck Rack found that the journalists PR teams pitch and the ones AI ends up citing overlap by just 2%, and about why identical prompts return different sources in AI visibility measurement and sampling noise.

Set the policy, do not inherit it

Blocking Training while allowing Agent is a defensible position. You keep your corpus out of the next training run and you still show up when a person asks a chatbot about you. Blocking everything is also defensible, and it costs visibility. Both are real choices.

Inheriting a default is not a choice.

So before Tuesday, check which of your domains sit on the free tier, confirm whether your pages count as ad-supported under Cloudflare’s definition (a remarketing pixel is easy to forget), and if anyone has already enabled a Training block, open Search Console today and verify Googlebot is still crawling. Do not wait for the traffic graph to tell you.

At IQ Source this is the first hour of an AEO engagement: decide the policy per category, write it into robots.txt and Cloudflare so both agree, then stand up measurement that survives the sampling noise. The date is fixed either way. What is still open is who in your company gets to make the call.

Set your crawler policy before Tuesday