Cloudflare
Last reviewed
Sources
31 citations
Review status
Source-backed
Revision
v2 · 2,653 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Sources
31 citations
Review status
Source-backed
Revision
v2 · 2,653 words
Add missing citations, update stale details, or suggest a clearer explanation.
Cloudflare, Inc. is an American internet infrastructure company that has become one of the most consequential players in the AI economy by running serverless AI inference at the network edge and by setting the terms of the fight over AI web crawling. Its network handles traffic for roughly 20 percent of all websites from data centers in more than 330 cities, which gives a single product default at Cloudflare unusual reach over how a fifth of the web behaves toward AI systems [1]. Founded in 2009 and long known for content delivery and security, Cloudflare now matters to AI on two fronts. As a developer platform, it runs serverless GPU inference (Workers AI), a vector database (Vectorize), and an AI API proxy (AI Gateway) across its edge computing network, and hosts AI agents and Model Context Protocol servers. As an intermediary, it leads the infrastructure-level fight over AI crawling: on July 1, 2025 it became the first major internet infrastructure provider to block AI crawlers by default and launched Pay Per Crawl, a marketplace in which AI companies pay publishers for access to content [2][3].
Cloudflare sits as a reverse proxy between visitors and the websites it protects, which gives its product defaults unusual reach: a setting change at Cloudflare instantly alters how a fifth of the web behaves toward web crawlers, attackers, and now AI systems. The company reports data centers in over 330 cities in more than 120 countries, peering with more than 12,500 networks, and total network capacity in excess of 500 terabits per second [1][4][29]. It has been listed on the New York Stock Exchange under the ticker NET since September 2019 [5]. Revenue for fiscal year 2025 was $2.17 billion, up 30 percent year over year, with fourth-quarter revenue of $614.5 million growing 34 percent; in that quarter the company closed its largest deal ever, averaging $42.5 million in annual contract value, after starting the year with a $130 million, five-year contract that was its largest by total value to date [6][30]. Co-founder Matthew Prince serves as chief executive and co-founder Michelle Zatlyn as president.
Cloudflare grew out of Project Honey Pot, an anti-spam tracking system that Prince and Lee Holloway created in 2004. Prince, Holloway, and Zatlyn founded the company in 2009, the same year the idea won the Harvard Business School business plan competition, and launched the service publicly at TechCrunch Disrupt on September 27, 2010 [5]. The company expanded from CDN and DDoS mitigation into enterprise security and, with the 2017 introduction of Cloudflare Workers, into serverless computing at the edge.
Its AI era began in earnest at the company's September 2023 "Birthday Week," when it launched Workers AI, Vectorize, and AI Gateway as a full-stack platform for AI applications [7]. From mid-2024 the company moved progressively against unauthorized AI crawling, culminating in the July 2025 default block and Pay Per Crawl. Prince has described building a content marketplace for the AI era as the company's "fourth act," following speed and security for websites, enterprise security, and the developer platform [8].
Workers AI, announced on September 27, 2023, lets developers run open-weight models serverlessly on GPUs deployed inside Cloudflare's own network, billed per request rather than by rented hardware [9]. A partnership announced the same day put NVIDIA GPUs and Ethernet switching into Cloudflare's points of presence; the service launched in seven sites, with roughly 100 planned by the end of 2023, and inference was available in more than 100 cities by year end [10]. By September 2024 the company said GPUs were live in more than 180 cities, and its product pages later advertised more than 190 locations [11]. The launch catalog emphasized curated open models, including Meta's Llama 2 and OpenAI's Whisper speech recognizer, with a Hugging Face partnership for deploying models from its hub [9].
| Product | Launched | Function |
|---|---|---|
| Workers AI | September 2023 | Serverless GPU inference on open models, billed per request [9] |
| Vectorize | September 2023, GA September 2024 | Vector database for embeddings and retrieval-augmented generation [7][11] |
| AI Gateway | September 2023 beta | Unified proxy for AI APIs with caching, rate limiting, logging, and cost analytics [12] |
| Agents SDK | February 2025 | Framework for stateful AI agents built on Durable Objects [13] |
| Remote MCP hosting | March 2025 | Deployment of remote Model Context Protocol servers with OAuth [14] |
AI Gateway acts as a control plane between applications and model providers, adding observability, caching, and rate limiting to third-party APIs as well as Workers AI [12]. Vectorize stores embeddings close to users so retrieval queries run in the same locations as inference [7]. Together these products make Cloudflare a distribution layer for AI infrastructure that places models near end users rather than in a handful of centralized data centers.
In July 2024 Cloudflare gave every customer, including free plans, a single toggle to block known AI scrapers and crawlers; more than one million customers had enabled it within a year [2][15]. In September 2024 it added AI Audit, later renamed AI Crawl Control, an analytics view showing which AI companies crawl a site, how often, and for what purpose [16].
In March 2025 the company introduced AI Labyrinth, an opt-in defense that responds to misbehaving crawlers not by blocking them but by luring them into a maze of AI-generated decoy pages. The links are hidden from human visitors and flagged so that legitimate search crawlers ignore them; bots that follow them waste compute on irrelevant content and fingerprint themselves, since, as the company put it, "no real human would go four links deep into a maze of AI-generated nonsense" [17].
On July 1, 2025, a date Prince branded "Content Independence Day," Cloudflare changed the default so that every new domain on its network blocks AI crawlers unless the site owner grants permission, and asked crawler operators to declare whether their bots fetch content for training, inference, or search [2][3]. Prince argued the traffic bargain underpinning the open web had collapsed, writing that "the web is being stripmined by AI crawlers with content creators seeing almost no traffic" [3]. By Cloudflare's data it had become roughly ten times harder to earn a visitor from Google crawling than a decade earlier, 750 times harder via OpenAI, and 30,000 times harder via Anthropic, as answer engines increasingly consume content without sending readers to its source [3].
Alongside the default change, Cloudflare opened a private beta of Pay Per Crawl, which repurposes the long-dormant HTTP 402 "Payment Required" status code. Publishers set a flat per-request price and choose, per crawler, whether to allow, charge, or block; a paying crawler either presents payment intent headers up front and receives the content (HTTP 200) or receives a 402 response with the price and retries, with Cloudflare acting as "the Merchant of Record for pay per crawl" and settling payments between AI companies and publishers [18]. The stated goal, in Cloudflare's words, is to "empower content owners to monetize their content at Internet scale" [18]. Coverage at the time noted that, because Cloudflare fronts about 20 percent of the web, the move amounted to a unilateral rewrite of crawling norms, and large publishers including The Associated Press, The Atlantic, Condé Nast, and BuzzFeed publicly backed it [2][19][20].
| Date | Step |
|---|---|
| July 2024 | Free one-click block of AI scrapers for all customers [15] |
| September 2024 | AI Audit / AI Crawl Control analytics [16] |
| March 2025 | AI Labyrinth decoy-page honeypot [17] |
| July 1, 2025 | Default blocking for new domains; Pay Per Crawl private beta [2][18] |
| August 2025 | Perplexity delisted as a verified bot over stealth crawling [21] |
| September 2025 | Content Signals Policy; x402 Foundation; NET Dollar plans [22][24][25] |
| January 2026 | Acquisition of AI data marketplace Human Native [8] |
In August 2025 Cloudflare publicly accused Perplexity of using "stealth, undeclared crawlers" to evade blocks: when its declared bots PerplexityBot and Perplexity-User were refused, requests allegedly resumed from a generic Chrome-on-macOS user agent across rotating IP addresses and networks outside Perplexity's published ranges, observed on tens of thousands of domains at millions of requests per day. Cloudflare delisted Perplexity as a verified bot and deployed rules to block the behavior [21]. Perplexity rejected the report as a publicity stunt built on "embarrassing errors," saying Cloudflare had misattributed three to six million daily requests from BrowserBase, a third-party cloud browser service, and arguing that user-driven assistants fetching a page on request are not crawlers in the traditional sense [23]. The episode sharpened an unresolved question in the crawler fight: whether agents acting for individual users should be treated like training scrapers.
On September 24, 2025 Cloudflare published the Content Signals Policy, an extension to robots.txt that lets sites state machine-readable preferences along three dimensions: "search," "ai-input" (use in answers, as in AI Overviews or RAG), and "ai-train." Cloudflare added the language automatically for the more than 3.8 million domains using its managed robots.txt, defaulting them to refuse training use while leaving search set to yes and ai-input neutral [22]. Publisher groups welcomed the tool, though trade coverage noted robots.txt signals remain unenforceable preferences and that the policy is partly aimed at Google, whose single crawler bundles search indexing with AI uses [26].
That same week Cloudflare and Coinbase announced the x402 Foundation, promoting an HTTP 402-based open standard for machine-to-machine payments, with x402 support in Cloudflare's agent tooling [24]; Cloudflare also announced plans for NET Dollar, a US dollar-backed stablecoin intended for instant agentic transactions [25]. On January 15, 2026 it acquired Human Native, a UK-based marketplace founded in 2024 and backed by LocalGlobe and Mercuri that licenses creator content to AI developers, saying the technology would extend Pay Per Crawl into a fuller content marketplace where AI companies can discover, license, and pay for data [8][31].
Cloudflare has positioned Workers as a natural substrate for AI agents because Durable Objects give each agent instance persistent state, an embedded SQLite database, WebSockets, and scheduling at near-zero idle cost. The Agents SDK, an open-source framework released in February 2025, packages these primitives, alongside Workers AI updates such as tool calling and structured outputs [13]. In March 2025 Cloudflare became the first platform to support deploying remote Model Context Protocol servers, adding an McpAgent class that handles transport and OAuth-based authorization so that clients such as Claude can connect to tools over the internet rather than via local processes [14]. The company runs its own catalog of managed MCP servers and has integrated x402 into the SDK for agent-initiated payments [24].
On November 18, 2025 Cloudflare suffered its worst outage since 2019, taking down or degrading ChatGPT, Claude, X, Shopify, Indeed, and many other services for several hours from 11:20 UTC [27][28]. The cause was internal, not an attack: a database permissions change made a query return duplicate rows, doubling the size of a "feature file" consumed by the Bot Management system, the machine-learning component that scores crawlers and bots. The file grew from about 60 features to more than 200, exceeding a hard 200-feature limit in the core proxy software, which then panicked across the network; core traffic largely recovered by 14:30 UTC after engineers rolled back to a known-good file, with full restoration at 17:06 UTC [27]. The incident underscored both how much of the AI ecosystem depends on Cloudflare and the centrality of the very bot-classification machinery at the heart of its AI crawler strategy.
Cloudflare's scale makes it a de facto policy-setter for the open web's relationship with AI. Its 2025 shift to permission-based crawling was the first time a major infrastructure provider, rather than individual publishers or courts, changed the economics of scraping by default, and its 402-based payment experiments, x402 standardization work, and Human Native acquisition sketch an alternative business model in which AI companies pay for content rather than harvest it [2][19][24]. The approach has critics: publishers have argued the signals lack teeth against determined scrapers, AI companies dispute Cloudflare's bot attributions, and concentrating crawler governance in one private gateway raises its own questions [23][26]. On the developer side, Workers AI, AI Gateway, and the Agents SDK have made Cloudflare a significant distribution layer for inference and agentic applications, competing with hyperscale clouds by placing AI close to users in over 190 cities [11][13].