Cloudflare
Last reviewed
Jun 9, 2026
Sources
28 citations
Review status
Source-backed
Revision
v1 ยท 2,397 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Jun 9, 2026
Sources
28 citations
Review status
Source-backed
Revision
v1 ยท 2,397 words
Add missing citations, update stale details, or suggest a clearer explanation.
Cloudflare, Inc. is an American internet infrastructure company that operates one of the world's largest edge networks, handling traffic for roughly 20 percent of all websites from data centers in more than 330 cities [1]. Founded in 2009 and long known for content delivery and security services, Cloudflare has become one of the most consequential companies in the AI economy along two fronts. As a developer platform, it runs serverless GPU inference (Workers AI), a vector database (Vectorize), and an AI API proxy (AI Gateway) across its network, and hosts AI agents and Model Context Protocol servers. As an intermediary, it leads the infrastructure-level fight over AI crawling: on July 1, 2025 it became the first major internet infrastructure provider to block AI crawlers by default and launched Pay Per Crawl, a marketplace in which AI companies pay publishers for access to content [2][3].
Cloudflare sits as a reverse proxy between visitors and the websites it protects, which gives its product defaults unusual reach: a setting change at Cloudflare instantly alters how a fifth of the web behaves toward crawlers, attackers, and now AI systems. The company reports data centers in over 330 cities in more than 120 countries and network capacity in excess of 500 terabits per second [1][4]. It has been listed on the New York Stock Exchange under the ticker NET since September 2019 [5]. Revenue for fiscal year 2025 was $2.17 billion, up 30 percent year over year, with fourth-quarter revenue of $614.5 million growing 34 percent; in that quarter the company closed its largest deal ever, averaging $42.5 million in annual contract value [6]. Co-founder Matthew Prince serves as chief executive and co-founder Michelle Zatlyn as president.
Cloudflare grew out of Project Honey Pot, an anti-spam tracking system that Prince and Lee Holloway created in 2004. Prince, Holloway, and Zatlyn founded the company in 2009, the same year the idea won the Harvard Business School business plan competition, and launched the service publicly at TechCrunch Disrupt on September 27, 2010 [5]. The company expanded from CDN and DDoS mitigation into enterprise security and, with the 2017 introduction of Cloudflare Workers, into serverless computing at the edge.
Its AI era began in earnest at the company's September 2023 "Birthday Week," when it launched Workers AI, Vectorize, and AI Gateway as a full-stack platform for AI applications [7]. From mid-2024 the company moved progressively against unauthorized AI crawling, culminating in the July 2025 default block and Pay Per Crawl. Prince has described building a content marketplace for the AI era as the company's "fourth act," following speed and security for websites, enterprise security, and the developer platform [8].
Workers AI, announced on September 27, 2023, lets developers run open-weight models serverlessly on GPUs deployed inside Cloudflare's own network, billed per request rather than by rented hardware [9]. A partnership announced the same day put NVIDIA GPUs and Ethernet switching into Cloudflare's points of presence, with inference available in more than 100 cities by the end of 2023 [10]. By September 2024 the company said GPUs were live in more than 180 cities, and its product pages later advertised more than 190 locations [11]. The launch catalog emphasized curated open models, including Meta's Llama 2 and OpenAI's Whisper speech recognizer, with a Hugging Face partnership for deploying models from its hub [9].
| Product | Launched | Function |
|---|---|---|
| Workers AI | September 2023 | Serverless GPU inference on open models, billed per request [9] |
| Vectorize | September 2023, GA September 2024 | Vector database for embeddings and retrieval-augmented generation [7][11] |
| AI Gateway | September 2023 beta | Unified proxy for AI APIs with caching, rate limiting, logging, and cost analytics [12] |
| Agents SDK | February 2025 | Framework for stateful AI agents built on Durable Objects [13] |
| Remote MCP hosting | March 2025 | Deployment of remote Model Context Protocol servers with OAuth [14] |
AI Gateway acts as a control plane between applications and model providers, adding observability, caching, and rate limiting to third-party APIs as well as Workers AI [12]. Vectorize stores embeddings close to users so retrieval queries run in the same locations as inference [7].
In July 2024 Cloudflare gave every customer, including free plans, a single toggle to block known AI scrapers and crawlers; more than one million customers had enabled it within a year [2][15]. In September 2024 it added AI Audit, later renamed AI Crawl Control, an analytics view showing which AI companies crawl a site, how often, and for what purpose [16].
In March 2025 the company introduced AI Labyrinth, an opt-in defense that responds to misbehaving crawlers not by blocking them but by luring them into a maze of AI-generated decoy pages. The links are hidden from human visitors and flagged so that legitimate search crawlers ignore them; bots that follow them waste compute on irrelevant content and fingerprint themselves, since, as the company put it, "no real human would go four links deep into a maze of AI-generated nonsense" [17].
On July 1, 2025, a date Prince branded "Content Independence Day," Cloudflare changed the default so that every new domain on its network blocks AI crawlers unless the site owner grants permission, and asked crawler operators to declare whether their bots fetch content for training, inference, or search [2][3]. Prince argued the traffic bargain underpinning the open web had collapsed: by Cloudflare's data it had become ten times harder to earn a visitor from Google crawling than a decade earlier, 750 times harder via OpenAI, and 30,000 times harder via Anthropic, as answer engines increasingly consume content without sending readers to its source [3].
Alongside the default change, Cloudflare opened a private beta of Pay Per Crawl, which repurposes the long-dormant HTTP 402 "Payment Required" status code. Publishers set a flat per-request price and choose, per crawler, whether to allow, charge, or block; a paying crawler either presents payment intent headers up front or receives a 402 response with the price and retries, with Cloudflare acting as merchant of record and settling payments between AI companies and publishers [18]. Coverage at the time noted that, because Cloudflare fronts about 20 percent of the web, the move amounted to a unilateral rewrite of crawling norms, and large publishers publicly backed it [19][20].
| Date | Step |
|---|---|
| July 2024 | Free one-click block of AI scrapers for all customers [15] |
| September 2024 | AI Audit / AI Crawl Control analytics [16] |
| March 2025 | AI Labyrinth decoy-page honeypot [17] |
| July 1, 2025 | Default blocking for new domains; Pay Per Crawl private beta [2][18] |
| August 2025 | Perplexity delisted as a verified bot over stealth crawling [21] |
| September 2025 | Content Signals Policy; x402 Foundation; NET Dollar plans [22][24][25] |
| January 2026 | Acquisition of AI data marketplace Human Native [8] |
In August 2025 Cloudflare publicly accused Perplexity of using "stealth, undeclared crawlers" to evade blocks: when its declared bots PerplexityBot and Perplexity-User were refused, requests allegedly resumed from a generic Chrome-on-macOS user agent across rotating IP addresses and networks outside Perplexity's published ranges, observed on tens of thousands of domains at millions of requests per day. Cloudflare delisted Perplexity as a verified bot and deployed rules to block the behavior [21]. Perplexity rejected the report as a publicity stunt built on "embarrassing errors," saying Cloudflare had misattributed three to six million daily requests from BrowserBase, a third-party cloud browser service, and arguing that user-driven assistants fetching a page on request are not crawlers in the traditional sense [23]. The episode sharpened an unresolved question in the crawler fight: whether agents acting for individual users should be treated like training scrapers.
On September 24, 2025 Cloudflare published the Content Signals Policy, an extension to robots.txt that lets sites state machine-readable preferences along three dimensions: "search," "ai-input" (use in answers, as in AI Overviews or RAG), and "ai-train." Cloudflare added the language automatically for the more than 3.8 million domains using its managed robots.txt, defaulting them to refuse training use [22]. Publisher groups welcomed the tool, though trade coverage noted robots.txt signals remain unenforceable preferences and that the policy is partly aimed at Google, whose single crawler bundles search indexing with AI uses [26].
That same week Cloudflare and Coinbase announced the x402 Foundation, promoting an HTTP 402-based open standard for machine-to-machine payments, with x402 support in Cloudflare's agent tooling [24]; Cloudflare also announced plans for NET Dollar, a US dollar-backed stablecoin intended for instant agentic transactions [25]. On January 15, 2026 it acquired Human Native, a UK-based marketplace that licenses creator content to AI developers, saying the technology would extend Pay Per Crawl into a fuller content marketplace where AI companies can discover, license, and pay for data [8].
Cloudflare has positioned Workers as a natural substrate for AI agents because Durable Objects give each agent instance persistent state, an embedded SQLite database, WebSockets, and scheduling at near-zero idle cost. The Agents SDK, an open-source framework released in February 2025, packages these primitives, alongside Workers AI updates such as tool calling and structured outputs [13]. In March 2025 Cloudflare became the first platform to support deploying remote Model Context Protocol servers, adding an McpAgent class that handles transport and OAuth-based authorization so that clients such as Claude can connect to tools over the internet rather than via local processes [14]. The company runs its own catalog of managed MCP servers and has integrated x402 into the SDK for agent-initiated payments [24].
On November 18, 2025 Cloudflare suffered its worst outage since 2019, taking down or degrading ChatGPT, Claude, X, Shopify, Indeed, and many other services for several hours from 11:20 UTC [27][28]. The cause was internal, not an attack: a database permissions change made a query return duplicate rows, doubling the size of a "feature file" consumed by the Bot Management system, the machine-learning component that scores crawlers and bots. The file exceeded a hard 200-feature limit in the core proxy software, which then panicked across the network; core traffic largely recovered by 14:30 UTC after engineers rolled back to a known-good file, with full restoration at 17:06 UTC [27]. The incident underscored both how much of the AI ecosystem depends on Cloudflare and the centrality of the very bot-classification machinery at the heart of its AI crawler strategy.
Cloudflare's scale makes it a de facto policy-setter for the open web's relationship with AI. Its 2025 shift to permission-based crawling was the first time a major infrastructure provider, rather than individual publishers or courts, changed the economics of scraping by default, and its 402-based payment experiments, x402 standardization work, and Human Native acquisition sketch an alternative business model in which AI companies pay for content rather than harvest it [2][19][24]. The approach has critics: publishers have argued the signals lack teeth against determined scrapers, AI companies dispute Cloudflare's bot attributions, and concentrating crawler governance in one private gateway raises its own questions [23][26]. On the developer side, Workers AI, AI Gateway, and the Agents SDK have made Cloudflare a significant distribution layer for inference and agentic applications, competing with hyperscale clouds by placing AI close to users in over 190 cities [11][13].