AI Wiki

Web scraping

AI InfrastructureAI Policy & RegulationData & Datasets
17 min read
Updated
Suggest editHistoryTalk24 citationsFact-checked
RawGraph

Web scraping is the automated extraction of data from websites, performed by programs that request pages the way a browser does and then parse the returned HTML, or the APIs behind it, into structured records [1]. Its close relative, web crawling, is the systematic traversal of the web by following hyperlinks from page to page: crawlers discover and download pages at scale, while scrapers pull specific fields out of them, and most large collection systems do both [1][2]. The techniques are as old as the search engine industry itself and have long powered price comparison, market research, journalism, and academic studies.

Since the rise of large language models, scraped web text has become the single largest raw material for AI training. Sixty percent of GPT-3's weighted pre-training data came from a filtered version of the Common Crawl web archive [11][12], and nearly every major open pretraining corpus, including C4, The Pile, RefinedWeb, Dolma, and FineWeb, is built from crawled pages. That demand turned what had been a niche dispute between platforms and data startups into a structural fight over who may use the open web and on what terms. Publishers now block AI crawlers at rates never seen for search crawlers, infrastructure providers sell crawler blocking and per-crawl payment, and courts are testing scraping under anti-hacking, contract, and copyright law [4][13][19].

The legal picture is unsettled and differs by jurisdiction. In the United States, the Ninth Circuit's hiQ v. LinkedIn rulings put scraping of publicly accessible pages largely outside the Computer Fraud and Abuse Act, while contract and copyright claims remain open [14]. The European Union permits commercial text and data mining only where rightsholders have not opted out by machine-readable means, and its AI Act obliges model providers to honor those reservations [17][18].

Techniques

Scraping systems range from single scripts to distributed platforms, but the underlying approaches fall into a few families [1]:

  • Direct HTTP requests. The scraper fetches raw HTML or JSON endpoints with an HTTP client and extracts fields with an HTML parser or pattern matching. This is cheap and fast but fails on pages that only render their content with JavaScript.
  • Headless browsers. The scraper drives a real browser engine without a visible window, using automation frameworks such as Selenium or Playwright, waits for the page's scripts to run, and then reads the resulting document object model [1]. This handles modern single-page applications at a much higher cost per page.
  • Wrapper and template extraction. For sites generated from databases, programs detect the shared page template and translate each page into relational rows [1].

Sites that do not want to be scraped answer with layered countermeasures: IP blocking and rate limiting, CAPTCHAs, TLS and browser fingerprinting, honeypot links invisible to human visitors, and deliberately poisoned "tarpit" content meant to waste a crawler's resources or contaminate its output [1]. Commercial scraping operations counter with rotating proxy networks, fingerprint spoofing, and CAPTCHA-solving services. Neither side has a durable technical advantage, which is one reason the conflict keeps moving into contracts, courts, and payment schemes.

Web crawlers

A crawler starts from a set of seed URLs, downloads those pages, extracts the hyperlinks they contain, and appends new URLs to a queue called the crawl frontier, repeating until a stopping condition is met [2]. Production crawlers are defined by four policies: a selection policy that decides which pages to fetch first (breadth-first order or importance measures such as PageRank), a re-visit policy that balances freshness against crawl cost, a politeness policy that spaces requests to any one server, and a parallelization policy that splits the frontier across machines without duplicating work [2]. URL normalization and content deduplication keep the same resource from being processed repeatedly [2]. Well-behaved crawlers identify themselves through the HTTP User-Agent header so that site operators can recognize and contact them; malicious crawlers routinely mask their identity [2].

robots.txt

The Robots Exclusion Protocol is the web's main opt-out mechanism for crawlers. Martijn Koster proposed it in February 1994, major search crawlers adopted it within months, and the IETF standardized it as RFC 9309 in September 2022 [3][4]. A site publishes a plain-text file at /robots.txt listing user agents and the paths they may not fetch. The protocol is purely advisory. RFC 9309 states that its rules "are not a form of access authorization," and nothing in the file can technically prevent a non-compliant crawler from fetching pages [3][4]. That gap between convention and enforcement sat mostly dormant during the search era, when the crawl-for-traffic exchange gave both sides a reason to cooperate, and it became the central fault line once AI training entered the picture.

Common Crawl

Common Crawl, a 501(c)(3) nonprofit founded by Gil Elbaz in 2007, has published free web crawls since 2011 and is the most important single source of crawled data for AI research [5][6]. Its corpus spans more than 300 billion pages collected over 15 years, grows by 3 to 5 billion pages each month via its CCBot crawler, and has been cited in over 10,000 research papers [5]. Filtered derivatives of Common Crawl form the backbone of most public pretraining datasets, and in 2023 OpenAI and Anthropic each donated $250,000 to the organization [6]. The archive's new prominence has a cost: as of 2025, CCBot was the most widely blocked bot among the top 1,000 websites [6]. Blocking CCBot removes a site from future snapshots and, indirectly, from every dataset later built on them.

AI crawlers

Since 2023 the major AI companies have documented their own crawler fleets, split by purpose into training crawlers, search-index crawlers, and user-triggered fetchers. OpenAI operates GPTBot to collect training data for its foundation models, OAI-SearchBot to build the index behind ChatGPT search, and ChatGPT-User for fetches made when a user asks ChatGPT to visit a specific page; each responds separately to robots.txt, though OpenAI notes that robots rules may not apply to user-initiated fetches [7]. Anthropic mirrors the structure with ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for live requests from Claude users, and states that its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt" [8]. Google took a different route: on September 28, 2023 it announced Google-Extended, a robots.txt control that lets publishers opt out of having their content improve its generative models and Vertex AI APIs without affecting Google Search indexing [9]. The distinction between training crawls, search indexing, and per-query retrieval matters for publishers, because blocking the first has no effect on the other two unless each token is blocked as well.

TokenOperatorPurposeSource
GPTBotOpenAITraining data for foundation models[7]
OAI-SearchBotOpenAIChatGPT search index[7]
ChatGPT-UserOpenAIUser-initiated page fetches[7]
ClaudeBotAnthropicTraining data collection[8]
Claude-SearchBotAnthropicSearch result indexing[8]
Claude-UserAnthropicUser-initiated page fetches[8]
CCBotCommon CrawlOpen web archive used widely in AI research[5][6]
Google-ExtendedGoogleOpt-out control for generative AI training[9]

Web data in the AI training pipeline

The path from crawl to model runs through heavy filtering. Google researchers built C4, the "Colossal Clean Crawled Corpus," by cleaning a Common Crawl snapshot for the T5 models in 2019 [10]. OpenAI's GPT-3 paper reported a filtered Common Crawl component of 410 billion tokens carrying 60 percent of the training weight, alongside WebText2, two book corpora, and Wikipedia [11][12]. Later open corpora such as The Pile, RefinedWeb, Dolma, and FineWeb refined the same recipe: language identification, deduplication, quality and toxicity filtering, and removal of benchmark contamination applied on top of raw crawl data. Image generators depend on the same infrastructure, since LAION's image-text datasets were assembled by mining crawled pages for image links and their alt text. Because scaling laws reward ever more data, the appetite for crawled text grew faster than the open web itself, pushing labs toward licensed archives and synthetic data, with model collapse studied as a risk of recycling model output back into pre-training.

The supply side is contracting. The Data Provenance Initiative's "Consent in Crisis" audit of 14,000 domains underlying C4, RefinedWeb, and Dolma found that in the single year from 2023 to 2024, new robots.txt restrictions fully restricted more than 5 percent of all tokens in C4 and more than 28 percent of its most actively maintained, critical sources, while terms of service restrictions covered around 45 percent of C4 [13]. The authors warned that if these signals are respected, the freshness and diversity of open training data will decline for commercial and academic builders alike [13].

No single law governs scraping. Disputes in the United States have been fought under the Computer Fraud and Abuse Act (CFAA), trespass to chattels, breach of contract, and copyright; in the European Union the key questions sit inside copyright's text and data mining exceptions.

CaseCourt and yearsOutcome
eBay v. Bidder's EdgeN.D. Cal., 2000Injunction against auction scraping under trespass to chattels [1]
Craigslist v. 3TapsN.D. Cal., 2013Cease-and-desist letter plus IP block held sufficient to support a CFAA claim [1]
hiQ Labs v. LinkedIn9th Cir., 2019 and 2022Scraping public profiles likely not a CFAA violation; hiQ later found to have breached LinkedIn's user agreement [14]
The New York Times v. Microsoft and OpenAIS.D.N.Y., filed 2023Pending; core copyright claims survived dismissal in March 2025 [15]
Bartz v. AnthropicN.D. Cal., 2024-2026Training on lawfully bought books held fair use; $1.5 billion settlement over pirated copies [16]

Computer access and contract law in the United States

The defining CFAA scraping case is hiQ Labs v. LinkedIn. hiQ, an analytics startup, scraped public LinkedIn profiles; LinkedIn sent a cease-and-desist and blocked its access; hiQ won a preliminary injunction in 2017, and the Ninth Circuit affirmed it on September 9, 2019 [14]. The Supreme Court vacated that ruling in June 2021 in light of Van Buren v. United States, which read the CFAA's "exceeds authorized access" clause narrowly, but on remand the Ninth Circuit reaffirmed on April 18, 2022 that scraping data available to the public without a login is not access "without authorization" under the CFAA, even after a cease-and-desist letter [14]. The victory was narrower than it first appeared: in November 2022 the district court found that hiQ had breached LinkedIn's User Agreement, and the parties settled [14]. The practical rule that emerged is that the federal anti-hacking statute is a poor weapon against public-data scraping, at least in the Ninth Circuit, while contract and terms-of-service claims remain live. Earlier cases point the same direction from different doctrines: eBay obtained an injunction against auction scraper Bidder's Edge in 2000 on a trespass-to-chattels theory, and a court let Craigslist's CFAA claim against 3Taps proceed in 2013 precisely because 3Taps kept scraping after an explicit revocation and IP block [1].

None of the access cases decided whether copying scraped content into training corpora infringes copyright, which is the question now being litigated in dozens of suits; see AI copyright. The most watched is The New York Times v. Microsoft and OpenAI, filed on December 27, 2023 in the Southern District of New York. The Times alleges that millions of its articles were used to train OpenAI's models without authorization, that its content was among the largest proprietary sources in the training data, and that ChatGPT and Copilot can produce "near-verbatim replicas" of its articles; it seeks damages and destruction of datasets containing its work [15]. OpenAI argues the training was transformative fair use and that the Times manipulated prompts to generate its regurgitation evidence [15]. On March 26, 2025, Judge Sidney Stein denied the motions to dismiss in large part, letting the central copyright claims proceed while dismissing several claims under the Digital Millennium Copyright Act, and the case was consolidated with suits by the New York Daily News and the Center for Investigative Reporting [15]. In June 2026 the Times moved to amend its complaint, adding allegations that Microsoft encouraged OpenAI's use of its articles while dropping its trademark dilution and contributory infringement claims [15]. As of mid-2026 there has been no merits ruling on fair use in the case.

The first major money judgment came out of a books case rather than a web-scraping one, but it reshaped how labs think about data acquisition. In Bartz v. Anthropic, filed in August 2024, Judge William Alsup ruled on June 23, 2025 that training on lawfully purchased and scanned books was fair use, while holding that Anthropic's downloading and retention of millions of pirated library copies from sources such as LibGen was not protected [16]. Anthropic then agreed in September 2025 to pay $1.5 billion, roughly $3,000 per book, the largest copyright settlement in US history, which received final court approval in July 2026 [16]. The ruling's logic makes how data was obtained, whether scraped from the open web, taken from pirate libraries, or licensed, a load-bearing part of the fair use analysis. Parallel suits, including Authors Guild v. OpenAI and the Perplexity copyright lawsuits, are still pending.

European Union text and data mining rules

The EU regulates scraping for AI mostly through copyright. Directive (EU) 2019/790 created two text and data mining (TDM) exceptions: Article 3 gives research organizations and cultural heritage institutions a mandatory exception for scientific research, and Article 4 extends TDM to everyone, including commercial AI developers, but only where rightsholders have not expressly reserved the use; for content made publicly available online, that reservation must be expressed by machine-readable means such as metadata or website terms [17]. The EU AI Act then bolted the opt-out onto AI regulation: Article 53(1)(c) requires providers of general-purpose AI models to "put in place a policy to comply with Union law on copyright and related rights, and in particular to identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790" [18]. The result is an opt-out regime: scraping for training is lawful by default in the EU, but a machine-readable reservation, unlike a US robots.txt file, carries direct legal weight.

Publisher blocking and the crawl economy

Publishers began blocking AI crawlers almost as soon as the crawlers were named. A 2023 analysis by Originality.AI found that 306 of the 1,000 most-visited websites blocked GPTBot and 85 blocked Google-Extended, with news organizations such as the BBC and The New York Times among the early blockers and Medium announcing a block on all AI crawlers [4]. The "Consent in Crisis" audit documented the same trend across whole training corpora within a year [13]. Enforcement remains the weak point: 404 Media reported that some AI-linked scrapers continued collecting from sites whose blocklists had not caught up with renamed agents [4], and on August 4, 2025 Cloudflare publicly accused Perplexity of using "stealth, undeclared crawlers" to reach sites that had blocked its declared bot, saying it rotated user agents and source networks and ignored, or sometimes never fetched, the robots.txt files of the sites involved; Cloudflare de-listed Perplexity as a verified bot in response [21].

Infrastructure providers have moved from documentation to defaults. On July 1, 2025, Cloudflare, which sits in front of a large share of web traffic, switched to blocking AI crawlers by default for new domains unless the site grants permission, arguing that the old crawl-for-traffic bargain had collapsed; the company said it had become roughly 750 times harder to earn a visitor from OpenAI's crawling, and roughly 30,000 times harder from Anthropic's, than from Google's traditional search crawl [19]. Alongside the default block it launched Pay Per Crawl in private beta, a scheme built on the HTTP 402 "Payment Required" status code in which authenticated crawlers see a per-request price, accept it through request headers, and are billed with Cloudflare acting as merchant of record, while publishers choose to allow, charge, or block each crawler individually [20]. On September 24, 2025 Cloudflare added a Content Signals Policy to the robots.txt files it manages for more than 3.8 million domains, expressing separate preferences for search indexing, AI input uses such as retrieval-augmented generation in AI search products, and AI training, with managed defaults of search yes and AI training no [22].

The other half of the emerging market is licensing: paying for what was once scraped for free. Reported deal terms are rarely official, but journalism trade coverage has tracked a steady escalation [23]. The New York Times itself signed an AI licensing deal with Amazon in May 2025 while its suit against OpenAI continued [15][23].

DealAnnouncedReported terms
Google and RedditFebruary 2024About $60 million per year for data licensing [24]
OpenAI and Dotdash MeredithMay 2024At least $16 million per year [23]
OpenAI and News CorpMay 2024More than $250 million over five years [23]
Amazon and The New York TimesMay 2025$20 million to $25 million per year [23]
Meta and News CorpMarch 2026Up to $50 million per year for at least three years [23]

Where this settles is genuinely open. The technical layer (robots.txt) is voluntary, the US legal layer is a patchwork still being litigated, and the EU layer makes opt-outs binding but depends on machine-readable signals that are only now being standardized. What has already changed is the default assumption of the search-era web: that anything publicly readable is also freely collectable.

See also

References

  1. ^Wikipedia: Web scraping. en.wikipedia.org/...Web_scraping
  2. ^Wikipedia: Web crawler. en.wikipedia.org/...Web_crawler
  3. ^Koster, M., Illyes, G., Zeller, H., Sassman, L. RFC 9309: Robots Exclusion Protocol. IETF, September 2022. rfc-editor.org/...rfc9309
  4. ^Wikipedia: robots.txt. en.wikipedia.org/...Robots.txt
  5. ^Common Crawl. Official website. commoncrawl.org
  6. ^Wikipedia: Common Crawl. en.wikipedia.org/...Common_Crawl
  7. ^OpenAI. "Overview of OpenAI crawlers and bots." OpenAI developer documentation. developers.openai.com/...bots
  8. ^Anthropic. "Does Anthropic crawl data from the web, and how can site owners block the crawler?" Anthropic Help Center. support.claude.com/...ite-owners-block-the-crawler
  9. ^Google. "An update on web publisher controls." The Keyword, September 28, 2023. blog.google/...an-update-on-web-publisher-controls
  10. ^Raffel, C. et al. "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." arXiv, October 2019. arxiv.org/...1910.10683
  11. ^Brown, T. et al. "Language Models are Few-Shot Learners." arXiv, May 2020. arxiv.org/...2005.14165
  12. ^Wikipedia: GPT-3. en.wikipedia.org/...GPT-3
  13. ^Longpre, S., Mahari, R. et al. "Consent in Crisis: The Rapid Decline of the AI Data Commons." arXiv, July 2024. arxiv.org/...2407.14933
  14. ^Wikipedia: hiQ Labs v. LinkedIn. en.wikipedia.org/...HiQ_Labs_v._LinkedIn
  15. ^Wikipedia: The New York Times v. Microsoft and OpenAI. en.wikipedia.org/..._Times_v._Microsoft_and_OpenAI
  16. ^Wikipedia: Bartz v. Anthropic. en.wikipedia.org/...Bartz_v._Anthropic
  17. ^Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market. EUR-Lex. eur-lex.europa.eu/...TXT
  18. ^EU Artificial Intelligence Act, Article 53: Obligations for Providers of General-Purpose AI Models. artificialintelligenceact.eu/...53
  19. ^Cloudflare. "Content Independence Day: no AI crawl without compensation!" Cloudflare Blog, July 1, 2025. blog.cloudflare.com/...-crawl-without-compensation
  20. ^Cloudflare. "Introducing pay per crawl: enabling content owners to charge AI crawlers for access." Cloudflare Blog, July 1, 2025. blog.cloudflare.com/introducing-pay-per-crawl
  21. ^Cloudflare. "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives." Cloudflare Blog, August 4, 2025. blog.cloudflare.com/...website-no-crawl-directives
  22. ^Cloudflare. "Giving users choice with Cloudflare's new Content Signals Policy." Cloudflare Blog, September 24, 2025. blog.cloudflare.com/content-signals-policy
  23. ^Press Gazette. "Who's suing AI and who's signing: news publisher deals and lawsuits with AI companies." pressgazette.co.uk/...deals-lawsuits-openai-google
  24. ^Wikipedia: Reddit (section on the February 2024 Google data licensing partnership). en.wikipedia.org/...Reddit

Improve this article

Add missing citations, update stale details, or suggest a clearer explanation. Every suggestion is reviewed for sourcing before it goes live.

v1 · 3,323 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent adversarial fact-check at creation (wanted38 campaign, 2026-07-24): every claim verified against primary sources by a dedicated verification agent; corrections applied before publication.

Cite this page: AI Wiki. "Web scraping." aiwiki.ai, updated 24 Jul 2026, fact-checked 24 Jul 2026. CC BY 4.0. https://aiwiki.ai/wiki/web_scraping

Suggest edit