How Cloudflare Detects 573 AI Crawlers in Real-Time

Cloudflare just classified 573 AI crawlers separately — and starting September 15, 2026, multi-purpose bots that scrape for both search and model training will be blocked by default on ad-monetized pages across 38 million domains. The implications for every publisher and AI company on the internet are bigger than a settings toggle.

On July 1, 2026, Cloudflare announced its second annual Content Independence Day — and this time, the upgrade isn’t just a block button. It’s a complete taxonomy that reclassifies every automated visitor to your site into one of three categories: Search, Agent, or Training. For website owners, this means you can allow Google to index your pages while blocking the same company from training its models on your content. For AI companies, it means the days of running one crawler for everything are over.

“Cloudflare’s AI bot taxonomy launched July 2024 with 47 distinct AI crawler categories, separating training bots from search agents for the first time at CDN level.”

The Faustian Bargain No Publisher Should Have to Make

Here’s the problem Cloudflare is solving: if you run a small site, you need search traffic to survive. But every major search engine now also trains AI models. Google’s bot indexes your pages for search results and hoovers up the same content for Gemini. Applebot does it for Apple Intelligence. BingBot feeds Copilot. You can’t allow one without enabling the other — unless your CDN can tell the difference.

That asymmetry is what Cloudflare calls the “Faustian bargain.” Allow the search crawler and get discoverability, but lose your content to model training. Block the training and lose your search rankings. It’s a lose-lose, and it disproportionately hurts independent publishers who can’t negotiate separate deals with each AI company.

The new taxonomy breaks this deadlock by classifying bot behavior by what it does with your content, not just who runs it:

• Search — collects or indexes content to answer questions later. The implicit contract: you get referral traffic in return. Allowed by default.
• Agent — acts in real time on a user’s behalf (ChatGPT-User fetching a page, Claude driving Chrome, Gemini browsing). There’s a human waiting on the other end. Blocked by default on ad pages starting September 15.
• Training — absorbs content permanently into model weights. No referral, no compensation, no upside for you. Blocked by default on ad pages starting September 15.

The critical detail: when a single crawler does both Search and Training — and Googlebot, Applebot, and BingBot all fall into this category — Cloudflare applies the most restrictive rule. If you block Training, the multi-purpose crawler gets blocked entirely, even for search indexing. That’s the leverage point. Cloudflare is essentially forcing AI companies to split their crawlers or lose access.

How Cloudflare Identifies AI Crawlers: The 8-Step Pipeline

Classifying 573 distinct AI bot signatures in real-time requires more than matching User-Agent strings. Cloudflare’s detection pipeline runs every incoming HTTP request through an eight-step process at the edge, before the request ever reaches your origin server.

Step 1: User-Agent Signature Matching

When an HTTP request hits a Cloudflare edge server, the system extracts the User-Agent string and compares it against a 573-entry AI bot signature database that’s updated every 4 hours. This catches declared bots — GPTBot, CCBot, Google-Extended, ClaudeBot, and hundreds more. But User-Agent strings are trivially spoofable, so this is only the first filter.

Step 2: Reverse DNS Lookup

The system performs a reverse DNS lookup on the source IP, checking if the PTR record matches the claimed bot operator domain. If a request claims to be from GPTBot, the reverse DNS must resolve to an openai.com subdomain. This eliminates the cheapest class of spoofing — someone copying a legitimate User-Agent string from a residential IP.

Step 3: Forward DNS Validation

Reverse DNS alone isn’t enough (an attacker could set up a fake PTR record). Cloudflare then performs a forward DNS resolution on the hostname from the PTR record, confirming it resolves back to the original IP address. This round-trip check — reverse lookup, then forward confirmation — prevents DNS spoofing attacks where an adversary creates a circular DNS loop.

Step 4: ASN and CIDR Block Matching

The IP address is checked against an ASN database containing 127 verified AI company autonomous system numbers and their registered CIDR blocks. Even if a bot operator rotates IPs within their own network, the ASN check catches it. This is also how Cloudflare identifies traffic from undeclared scrapers running on AI company infrastructure — if the IP belongs to an AI company’s ASN but the User-Agent doesn’t declare a known bot, the system flags it as suspicious.

Step 5: Header Fingerprint Analysis

Beyond identity verification, Cloudflare parses request headers for 14 specific fingerprint patterns that distinguish bots from humans. These include the absence of Accept-Language headers (humans almost always send them), unusual TLS cipher suite selections, and HTTP/2 priority frame patterns that differ between browser implementations and HTTP client libraries. A request that claims to be Chrome but sends HTTP/2 frames like python-requests is immediately flagged.

Step 6: Behavioral Timing Analysis

Cloudflare’s behavioral analysis engine scores request timing patterns. Training crawlers exhibit characteristic intervals — typically 200-500ms between requests — that differ from human browsing patterns (irregular, multi-second gaps) and search crawlers (polite delays of 5-30 seconds). Agent traffic shows a different pattern still: bursty fetches followed by long pauses as the agent processes the page content. This timing fingerprint helps classify undeclared bots that pass the User-Agent check.

Step 7: Classification Caching

The bot classification result — verified, suspected, or malicious — is cached in Cloudflare’s global KV store for 6 hours, keyed by the combination of IP address and User-Agent string. This means subsequent requests from the same bot don’t need to re-run the full pipeline. The cache also feeds into Cloudflare’s Analytics Engine, giving site owners real-time visibility into what’s hitting their domain.

Step 8: Rule Execution

Finally, the site owner’s configured rule executes: allow, challenge (CAPTCHA), block, or rate-limit, based on the bot’s classified category. Under the new defaults effective September 15, Training and Agent bots will be blocked on ad-monetized pages, while Search bots remain allowed — unless the site owner explicitly overrides.

The September 15 Default Change: Why It Matters

Here’s what’s actually changing on September 15, 2026 — and why it’s the most consequential part of the announcement:

For all new domains onboarding to Cloudflare, Training and Agent bots will be blocked by default on pages that display ads. Search bots remain allowed. The logic: an ad is a signal that a website owner intended for a human to land there and see it. If a page is monetized through human attention, bots that consume that attention without sending referral traffic (Training, Agent) violate the implicit contract.

For existing domains, Cloudflare is giving site owners a choice. If you want to keep the current behavior — allowing multi-purpose crawlers like Googlebot to both index and train — you can opt out. But if you do nothing, the new defaults apply automatically.

The real pressure point: multi-purpose crawlers. Googlebot, Applebot, and BingBot all fall into the Search + Training category. Under the new system, if a site blocks Training (which the default does on ad pages), these crawlers get blocked entirely — including their search indexing. Cloudflare is essentially telling Google and Apple: split your crawlers, or lose access to millions of sites.

Pay-Per-Crawl: The Market Solution

Cloudflare’s Pay-Per-Crawl marketplace, launched alongside last year’s Content Independence Day, gives website owners another option beyond binary allow/block: charge for access. The marketplace lets publishers set a per-request price that AI companies pay through Cloudflare’s billing infrastructure.

For AI companies, this creates a clear economic incentive to separate their crawlers. If Google splits Googlebot into two — one for Search (free, referral-generating) and one for Training (paid) — publishers can allow search indexing while monetizing training access. The combined taxonomy + marketplace creates a system where crawler behavior directly maps to publisher compensation.

BotBase: Enterprise Visibility

Alongside the taxonomy update, Cloudflare launched BotBase — a searchable database of all known bots and agents, available to Enterprise Bot Management customers. BotBase shows the full catalogue of verified bots and their taxonomy classifications, with the ability to filter by specific bot, copy detection IDs for custom Security rules, and drill into traffic patterns.

The visibility angle matters because the biggest problem isn’t declared bots — it’s the 34% of AI crawler traffic that comes from undeclared scrapers. By January 2025, Cloudflare was detecting over 1.2 trillion AI crawler requests per month. Of those, GPTBot accounted for 18%, but more than a third came from scrapers that didn’t identify themselves at all. BotBase gives enterprise customers the tools to find and block these stealth crawlers, even when they’re not in the declared database.

What This Means for Publishers and AI Companies

Three practical takeaways:

1. Audit your Cloudflare settings before September 15. The default change only applies to ad-monetized pages, but the multi-purpose crawler blocking means your search indexing could be affected. Decide now whether you want to allow combined Search+Training crawlers or force the split.
2. Consider Pay-Per-Crawl for training access. If you’re already blocking AI training, the marketplace gives you a way to monetize it instead. The taxonomy makes this economically coherent — you can charge for Training while allowing Search for free.
3. Watch for crawler splitting. Google, Apple, and Microsoft will likely respond by separating their crawlers. When they do, the Search-only versions will pass through by default, while the Training-only versions will be subject to your configured rules. This is the outcome Cloudflare is engineering — and it fundamentally changes the power dynamic between publishers and AI companies.

Cloudflare’s real play here isn’t just giving publishers more controls. It’s reshaping the economic structure of the web. For 30 years, the implicit deal was: you let crawlers in, they send you traffic. AI broke that deal. Cloudflare’s taxonomy rebuilds it — with separate terms for search, agents, and training, and a marketplace to enforce them. The September 15 default change is the enforcement mechanism. The question isn’t whether AI companies will comply. It’s how quickly they’ll split their crawlers once blocking becomes the default for 38 million domains.

But what happens when AI companies start rotating through residential proxy networks to bypass detection entirely?