# Cloudflare

> Source: https://aiwiki.ai/wiki/cloudflare
> Updated: 2026-06-23
> Categories: AI Companies, AI Infrastructure
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

Cloudflare, Inc. is an American internet infrastructure company that has become one of the most consequential players in the AI economy by running serverless AI [inference](/wiki/inference) at the network edge and by setting the terms of the fight over AI web crawling. Its network handles traffic for roughly 20 percent of all websites from data centers in more than 330 cities, which gives a single product default at Cloudflare unusual reach over how a fifth of the web behaves toward AI systems [1]. Founded in 2009 and long known for content delivery and security, Cloudflare now matters to AI on two fronts. As a developer platform, it runs serverless GPU inference (Workers AI), a [vector database](/wiki/vector_database) (Vectorize), and an AI API proxy (AI Gateway) across its [edge computing](/wiki/edge_computing) network, and hosts [AI agents](/wiki/ai_agent) and [Model Context Protocol](/wiki/model_context_protocol) servers. As an intermediary, it leads the infrastructure-level fight over AI crawling: on July 1, 2025 it became the first major internet infrastructure provider to block AI crawlers by default and launched Pay Per Crawl, a marketplace in which AI companies pay publishers for access to content [2][3].

## Overview

Cloudflare sits as a reverse proxy between visitors and the websites it protects, which gives its product defaults unusual reach: a setting change at Cloudflare instantly alters how a fifth of the web behaves toward [web crawlers](/wiki/web_crawler), attackers, and now AI systems. The company reports data centers in over 330 cities in more than 120 countries, peering with more than 12,500 networks, and total network capacity in excess of 500 terabits per second [1][4][29]. It has been listed on the New York Stock Exchange under the ticker NET since September 2019 [5]. Revenue for fiscal year 2025 was $2.17 billion, up 30 percent year over year, with fourth-quarter revenue of $614.5 million growing 34 percent; in that quarter the company closed its largest deal ever, averaging $42.5 million in annual contract value, after starting the year with a $130 million, five-year contract that was its largest by total value to date [6][30]. Co-founder [Matthew Prince](/wiki/matthew_prince) serves as chief executive and co-founder [Michelle Zatlyn](/wiki/michelle_zatlyn) as president.

## History

Cloudflare grew out of Project Honey Pot, an anti-spam tracking system that Prince and Lee Holloway created in 2004. Prince, Holloway, and Zatlyn founded the company in 2009, the same year the idea won the Harvard Business School business plan competition, and launched the service publicly at TechCrunch Disrupt on September 27, 2010 [5]. The company expanded from CDN and DDoS mitigation into enterprise security and, with the 2017 introduction of Cloudflare Workers, into serverless computing at the edge.

Its AI era began in earnest at the company's September 2023 "Birthday Week," when it launched Workers AI, Vectorize, and AI Gateway as a full-stack platform for AI applications [7]. From mid-2024 the company moved progressively against unauthorized AI crawling, culminating in the July 2025 default block and Pay Per Crawl. Prince has described building a content marketplace for the AI era as the company's "fourth act," following speed and security for websites, enterprise security, and the developer platform [8].

## What is Workers AI and the Cloudflare developer platform?

Workers AI, announced on September 27, 2023, lets developers run open-weight models serverlessly on GPUs deployed inside Cloudflare's own network, billed per request rather than by rented hardware [9]. A partnership announced the same day put [NVIDIA](/wiki/nvidia) GPUs and Ethernet switching into Cloudflare's points of presence; the service launched in seven sites, with roughly 100 planned by the end of 2023, and inference was available in more than 100 cities by year end [10]. By September 2024 the company said GPUs were live in more than 180 cities, and its product pages later advertised more than 190 locations [11]. The launch catalog emphasized curated open models, including [Meta](/wiki/meta_ai)'s [Llama 2](/wiki/llama_2) and [OpenAI](/wiki/openai)'s [Whisper](/wiki/whisper) speech recognizer, with a [Hugging Face](/wiki/hugging_face) partnership for deploying models from its hub [9].

| Product | Launched | Function |
| --- | --- | --- |
| Workers AI | September 2023 | Serverless GPU inference on open models, billed per request [9] |
| Vectorize | September 2023, GA September 2024 | Vector database for embeddings and [retrieval-augmented generation](/wiki/retrieval_augmented_generation) [7][11] |
| AI Gateway | September 2023 beta | Unified proxy for AI APIs with caching, rate limiting, logging, and cost analytics [12] |
| Agents SDK | February 2025 | Framework for stateful AI agents built on Durable Objects [13] |
| Remote MCP hosting | March 2025 | Deployment of remote Model Context Protocol servers with OAuth [14] |

AI Gateway acts as a control plane between applications and model providers, adding observability, caching, and rate limiting to third-party APIs as well as Workers AI [12]. Vectorize stores embeddings close to users so retrieval queries run in the same locations as inference [7]. Together these products make Cloudflare a distribution layer for [AI infrastructure](/wiki/ai_infrastructure) that places models near end users rather than in a handful of centralized data centers.

## How did the AI crawler fight unfold?

### From one-click blocking to AI Labyrinth

In July 2024 Cloudflare gave every customer, including free plans, a single toggle to block known AI scrapers and crawlers; more than one million customers had enabled it within a year [2][15]. In September 2024 it added AI Audit, later renamed AI Crawl Control, an analytics view showing which AI companies crawl a site, how often, and for what purpose [16].

In March 2025 the company introduced AI Labyrinth, an opt-in defense that responds to misbehaving crawlers not by blocking them but by luring them into a maze of AI-generated decoy pages. The links are hidden from human visitors and flagged so that legitimate search crawlers ignore them; bots that follow them waste compute on irrelevant content and fingerprint themselves, since, as the company put it, "no real human would go four links deep into a maze of AI-generated nonsense" [17].

### What is Content Independence Day and Pay Per Crawl?

On July 1, 2025, a date Prince branded "Content Independence Day," Cloudflare changed the default so that every new domain on its network blocks AI crawlers unless the site owner grants permission, and asked crawler operators to declare whether their bots fetch content for training, inference, or search [2][3]. Prince argued the traffic bargain underpinning the open web had collapsed, writing that "the web is being stripmined by AI crawlers with content creators seeing almost no traffic" [3]. By Cloudflare's data it had become roughly ten times harder to earn a visitor from [Google](/wiki/google) crawling than a decade earlier, 750 times harder via OpenAI, and 30,000 times harder via [Anthropic](/wiki/anthropic), as answer engines increasingly consume content without sending readers to its source [3].

Alongside the default change, Cloudflare opened a private beta of Pay Per Crawl, which repurposes the long-dormant HTTP 402 "Payment Required" status code. Publishers set a flat per-request price and choose, per crawler, whether to allow, charge, or block; a paying crawler either presents payment intent headers up front and receives the content (HTTP 200) or receives a 402 response with the price and retries, with Cloudflare acting as "the Merchant of Record for pay per crawl" and settling payments between AI companies and publishers [18]. The stated goal, in Cloudflare's words, is to "empower content owners to monetize their content at Internet scale" [18]. Coverage at the time noted that, because Cloudflare fronts about 20 percent of the web, the move amounted to a unilateral rewrite of crawling norms, and large publishers including The Associated Press, The Atlantic, Condé Nast, and BuzzFeed publicly backed it [2][19][20].

| Date | Step |
| --- | --- |
| July 2024 | Free one-click block of AI scrapers for all customers [15] |
| September 2024 | AI Audit / AI Crawl Control analytics [16] |
| March 2025 | AI Labyrinth decoy-page honeypot [17] |
| July 1, 2025 | Default blocking for new domains; Pay Per Crawl private beta [2][18] |
| August 2025 | Perplexity delisted as a verified bot over stealth crawling [21] |
| September 2025 | Content Signals Policy; x402 Foundation; NET Dollar plans [22][24][25] |
| January 2026 | Acquisition of AI data marketplace Human Native [8] |

### What was the Cloudflare-Perplexity dispute about?

In August 2025 Cloudflare publicly accused [Perplexity](/wiki/perplexity_ai) of using "stealth, undeclared crawlers" to evade blocks: when its declared bots PerplexityBot and Perplexity-User were refused, requests allegedly resumed from a generic Chrome-on-macOS user agent across rotating IP addresses and networks outside Perplexity's published ranges, observed on tens of thousands of domains at millions of requests per day. Cloudflare delisted Perplexity as a verified bot and deployed rules to block the behavior [21]. Perplexity rejected the report as a publicity stunt built on "embarrassing errors," saying Cloudflare had misattributed three to six million daily requests from BrowserBase, a third-party cloud browser service, and arguing that user-driven assistants fetching a page on request are not crawlers in the traditional sense [23]. The episode sharpened an unresolved question in the crawler fight: whether agents acting for individual users should be treated like training scrapers.

### Content Signals, payments, and Human Native

On September 24, 2025 Cloudflare published the Content Signals Policy, an extension to [robots.txt](/wiki/robots_txt) that lets sites state machine-readable preferences along three dimensions: "search," "ai-input" (use in answers, as in [AI Overviews](/wiki/ai_overviews) or RAG), and "ai-train." Cloudflare added the language automatically for the more than 3.8 million domains using its managed robots.txt, defaulting them to refuse training use while leaving search set to yes and ai-input neutral [22]. Publisher groups welcomed the tool, though trade coverage noted robots.txt signals remain unenforceable preferences and that the policy is partly aimed at Google, whose single crawler bundles search indexing with AI uses [26].

That same week Cloudflare and [Coinbase](/wiki/coinbase) announced the x402 Foundation, promoting an HTTP 402-based open standard for machine-to-machine payments, with x402 support in Cloudflare's agent tooling [24]; Cloudflare also announced plans for NET Dollar, a US dollar-backed stablecoin intended for instant agentic transactions [25]. On January 15, 2026 it acquired Human Native, a UK-based marketplace founded in 2024 and backed by LocalGlobe and Mercuri that licenses creator content to AI developers, saying the technology would extend Pay Per Crawl into a fuller content marketplace where AI companies can discover, license, and pay for data [8][31].

## Why is Cloudflare important for AI agents?

Cloudflare has positioned Workers as a natural substrate for AI agents because Durable Objects give each agent instance persistent state, an embedded SQLite database, WebSockets, and scheduling at near-zero idle cost. The Agents SDK, an open-source framework released in February 2025, packages these primitives, alongside Workers AI updates such as tool calling and structured outputs [13]. In March 2025 Cloudflare became the first platform to support deploying remote Model Context Protocol servers, adding an McpAgent class that handles transport and OAuth-based authorization so that clients such as [Claude](/wiki/claude) can connect to tools over the internet rather than via local processes [14]. The company runs its own catalog of managed MCP servers and has integrated x402 into the SDK for agent-initiated payments [24].

## November 2025 outage

On November 18, 2025 Cloudflare suffered its worst outage since 2019, taking down or degrading [ChatGPT](/wiki/chatgpt), Claude, X, Shopify, Indeed, and many other services for several hours from 11:20 UTC [27][28]. The cause was internal, not an attack: a database permissions change made a query return duplicate rows, doubling the size of a "feature file" consumed by the Bot Management system, the machine-learning component that scores crawlers and bots. The file grew from about 60 features to more than 200, exceeding a hard 200-feature limit in the core proxy software, which then panicked across the network; core traffic largely recovered by 14:30 UTC after engineers rolled back to a known-good file, with full restoration at 17:06 UTC [27]. The incident underscored both how much of the AI ecosystem depends on Cloudflare and the centrality of the very bot-classification machinery at the heart of its AI crawler strategy.

## Significance

Cloudflare's scale makes it a de facto policy-setter for the open web's relationship with AI. Its 2025 shift to permission-based crawling was the first time a major infrastructure provider, rather than individual publishers or courts, changed the economics of scraping by default, and its 402-based payment experiments, x402 standardization work, and Human Native acquisition sketch an alternative business model in which AI companies pay for content rather than harvest it [2][19][24]. The approach has critics: publishers have argued the signals lack teeth against determined scrapers, AI companies dispute Cloudflare's bot attributions, and concentrating crawler governance in one private gateway raises its own questions [23][26]. On the developer side, Workers AI, AI Gateway, and the Agents SDK have made Cloudflare a significant distribution layer for inference and agentic applications, competing with hyperscale clouds by placing AI close to users in over 190 cities [11][13].

## References

1. Cloudflare, "The Cloudflare global network." https://www.cloudflare.com/network/
2. Cloudflare press release, "Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large," July 1, 2025. https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/
3. Matthew Prince, "Content Independence Day: no AI crawl without compensation!" Cloudflare blog, July 1, 2025. https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/
4. Cloudflare blog, "500 Tbps of capacity: 16 years of scaling our global network," 2025. https://blog.cloudflare.com/500-tbps-of-capacity/
5. Wikipedia, "Cloudflare." https://en.wikipedia.org/wiki/Cloudflare
6. Cloudflare press release, "Cloudflare Announces Fourth Quarter and Fiscal Year 2025 Financial Results," February 10, 2026. https://www.cloudflare.com/press/press-releases/2026/cloudflare-announces-fourth-quarter-and-fiscal-year-2025-financial-results/
7. Cloudflare blog, "Birthday Week recap: everything we announced," September 2023. https://blog.cloudflare.com/birthday-week-2023-wrap-up/
8. Cloudflare blog, "Human Native is joining Cloudflare," January 15, 2026. https://blog.cloudflare.com/human-native-joins-cloudflare/
9. Cloudflare blog, "Workers AI: serverless GPU-powered inference on Cloudflare's global network," September 27, 2023. https://blog.cloudflare.com/workers-ai/
10. Cloudflare press release, "Cloudflare Powers Hyper-Local AI Inference with NVIDIA Accelerated Computing," September 27, 2023. https://www.cloudflare.com/press/press-releases/2023/cloudflare-powers-hyper-local-ai-inference-with-nvidia/
11. Cloudflare press release, "Cloudflare Enhances AI Inference Platform with Powerful GPU Upgrade, Faster Inference, Larger Models, Observability, and Upgraded Vector Database," September 2024. https://www.cloudflare.com/press/press-releases/2024/cloudflare-enhances-ai-inference-platform-with-powerful-gpu-upgrade/
12. Cloudflare blog, "Announcing AI Gateway: making AI applications more observable, reliable, and scalable," September 2023. https://blog.cloudflare.com/announcing-ai-gateway/
13. Cloudflare blog, "Making Cloudflare the best platform for building AI Agents," February 2025. https://blog.cloudflare.com/build-ai-agents-on-cloudflare/
14. Cloudflare blog, "Build and deploy Remote Model Context Protocol (MCP) servers to Cloudflare," March 2025. https://blog.cloudflare.com/remote-model-context-protocol-servers-mcp/
15. Cloudflare blog, "Declare your AIndependence: block AI bots, scrapers and crawlers with a single click," July 3, 2024. https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
16. Cloudflare blog, "Start auditing and controlling the AI models accessing your content," September 23, 2024. https://blog.cloudflare.com/cloudflare-ai-audit-control-ai-content-crawlers/
17. Cloudflare blog, "Trapping misbehaving bots in an AI Labyrinth," March 19, 2025. https://blog.cloudflare.com/ai-labyrinth/
18. Cloudflare blog, "Introducing pay per crawl: enabling content owners to charge AI crawlers for access," July 1, 2025. https://blog.cloudflare.com/introducing-pay-per-crawl/
19. MIT Technology Review, "Cloudflare will now, by default, block AI bots from crawling its clients' websites," July 1, 2025. https://www.technologyreview.com/2025/07/01/1119498/cloudflare-will-now-by-default-block-ai-bots-from-crawling-its-clients-websites/
20. Nieman Journalism Lab, "Cloudflare will block AI scraping by default and launches new 'Pay Per Crawl' marketplace," July 2025. https://www.niemanlab.org/2025/07/cloudflare-will-block-ai-scraping-by-default-and-launches-new-pay-per-crawl-marketplace/
21. Cloudflare blog, "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives," August 4, 2025. https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/
22. Cloudflare blog, "Giving users choice with Cloudflare's new Content Signals Policy," September 24, 2025. https://blog.cloudflare.com/content-signals-policy/
23. Slashdot, "Perplexity Says Cloudflare's Accusations of 'Stealth' AI Scraping Are Based On Embarrassing Errors," August 5, 2025. https://tech.slashdot.org/story/25/08/05/2153232/perplexity-says-cloudflares-accusations-of-stealth-ai-scraping-are-based-on-embarrassing-errors
24. Cloudflare blog, "Launching the x402 Foundation with Coinbase, and support for x402 transactions," September 2025. https://blog.cloudflare.com/x402/
25. Ledger Insights, "Cloudflare to launch NET stablecoin for AI agentic payments," September 2025. https://www.ledgerinsights.com/cloudflare-to-launch-net-stablecoin-for-ai-agentic-payments/
26. Digiday, "Cloudflare updates robots.txt for the AI era, but publishers still want more bite against bots," September 2025. https://digiday.com/media/cloudflare-updates-robots-txt-for-the-ai-era-but-publishers-still-want-more-bite-against-bots/
27. Cloudflare blog, "Cloudflare outage on November 18, 2025," November 2025. https://blog.cloudflare.com/18-november-2025-outage/
28. CNBC, "Cloudflare says outage that hit X, ChatGPT and other sites is resolved," November 18, 2025. https://www.cnbc.com/2025/11/18/cloudflare-down-outage-traffic-spike-x-chatgpt.html
29. Cloudflare blog, "500 Tbps of capacity: 16 years of scaling our global network," 2026. https://blog.cloudflare.com/500-tbps-of-capacity/
30. Cloudflare, Inc., "Q4 2025 Earnings Call," The Motley Fool transcript, February 10, 2026. https://www.fool.com/earnings/call-transcripts/2026/02/10/cloudflare-net-q4-2025-earnings-call-transcript/
31. CNBC, "Cloudflare acquires AI data marketplace Human Native," January 15, 2026. https://www.cnbc.com/2026/01/15/cloudflare-ai-human-native-acquisition.html