How to Prevent OpenAI and Google From Training Their LLMs on Your Website's Data

See also: Guides

You can stop OpenAI, Google, Anthropic, and most other major AI companies from using your website to train large language models (LLMs) by adding a small set of directives to your site's robots.txt file. The basic block is a few lines. Doing it well, across every bot that matters in 2026, takes a bit more care.

This guide covers the minimum viable block, the full list of AI crawlers worth disallowing, server level headers, page level meta tags, what to do if you are on Cloudflare, and a few notes on protecting images with Glaze and Nightshade. It also explains where these signals do not help, because the gap between "telling a bot to go away" and "actually preventing training" is wider than most people think.

the quick answer

If you only want to block the two crawlers in the article title, drop this into a file called robots.txt at the root of your domain (so it loads at https://yoursite.com/robots.txt):

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The first block tells OpenAI's training crawler, GPTBot, to skip your entire site. That prevents your pages from being added to the training data for future versions of ChatGPT, GPT-4, GPT-5, and any other OpenAI foundation models.

The second block tells Google's training token, Google-Extended, that your content should not be used to train Gemini (formerly Bard) or to ground Gemini Apps and Vertex AI generative APIs. Google introduced Google-Extended on September 28, 2023, and confirmed that it does not affect search ranking or indexing, only AI training and grounding.

If you want to block only part of your site, replace the / with a path:

User-agent: GPTBot
Disallow: /members/

User-agent: Google-Extended
Disallow: /members/

That is the quick version. It is also incomplete, because OpenAI and Google are not the only companies scraping the web for training data.

the full robots.txt for ai crawlers

In 2026 the list of bots actually doing training scraping is longer than it was when this article was first written. Here is a more complete block list, with each user agent named after the company that operates it.

# OpenAI
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

# Google
User-agent: Google-Extended
Disallow: /

# Anthropic
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

# Common Crawl (feeds many LLM training sets)
User-agent: CCBot
Disallow: /

# Perplexity
User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

# Apple Intelligence
User-agent: Applebot-Extended
Disallow: /

# ByteDance (TikTok parent, powers Doubao and other models)
User-agent: Bytespider
Disallow: /

# Meta AI
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: FacebookBot
Disallow: /

# Cohere
User-agent: cohere-ai
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

# Amazon
User-agent: Amazonbot
Disallow: /

A few things to know before pasting this in.

Training bots and search bots are not the same. OpenAI, Anthropic, Google, and Perplexity each split their crawling into separate user agents for separate purposes. GPTBot trains models; OAI-SearchBot powers ChatGPT search results; ChatGPT-User fetches a single page when a user clicks a link or asks ChatGPT to look something up. ClaudeBot trains Claude; Claude-SearchBot builds a search index; Claude-User makes per query fetches from Claude.ai. Blocking the training bot does not block the search bot, and vice versa. Many publishers block the training bots and allow the search bots so that their pages can still surface as cited links in AI search answers.

Google-Extended is unusual. It is not a separate crawler. Google still uses Googlebot to fetch your page; the Google-Extended token in robots.txt is a signal that says "crawled content from this site is not allowed to be used for Gemini training or grounding." Google has clarified that Google-Extended controls Gemini Apps and the Vertex AI generative APIs.

Some old user agents are no longer real. anthropic-ai and Claude-Web are deprecated. Anthropic has confirmed that sites blocking only those strings are not blocking its current ClaudeBot. If your existing robots.txt only mentions anthropic-ai, update it.

Common Crawl is the silent training pipeline. CCBot belongs to Common Crawl, a non profit that publishes a huge open repository of crawled web pages. By 2026 that archive contains more than 250 billion pages collected over more than 15 years, and it is the default starting point for most LLM training runs, including the early versions of GPT-3. Blocking CCBot is arguably more impactful than blocking any single company's bot.

Bytespider is aggressive. Bytespider, operated by ByteDance, was one of the most active AI scrapers in mid 2024, and publishers report it ignoring crawl delay directives. Block it explicitly.

meta tags and x-robots-tag headers

robots.txt is a site wide signal. If you want to set the rule per page, you can use the robots meta tag inside an HTML document:

<meta name="robots" content="noai, noimageai">

noai asks AI crawlers not to use the page's text, code, or structured data for model training. noimageai asks them not to use the page's images for training image models. These directives were originally proposed by DeviantArt in late 2022 and have been picked up unevenly. They are not part of the formal robots.txt standard, but several large AI companies have indicated they will honor them.

If you cannot edit HTML, for example because your pages are PDFs, JSON files, or static assets served by a CDN, set the same signal as an HTTP response header:

X-Robots-Tag: noai, noimageai

For an Nginx server you would add the header in the server block. On Apache you would set it in .htaccess. On Vercel, Netlify, or Cloudflare Workers you can attach it in the platform's headers configuration.

Meta tags and X-Robots-Tag are best treated as belt and braces, used alongside robots.txt rather than instead of it. Crawlers that ignore one tend to ignore the other.

blocking ai bots through cloudflare

If your site sits behind Cloudflare, you have a stronger option than robots.txt. On July 3, 2024 Cloudflare launched a one click toggle in the dashboard under Security and Bots labeled "AI Scrapers and Crawlers." Flipping it on tells Cloudflare's edge to block requests from a maintained list of known AI bots regardless of whether they obey robots.txt. The feature is free for every Cloudflare customer, including the free tier.

In July 2025 Cloudflare went further and announced that new domains signing up to Cloudflare would have AI crawler blocking enabled by default, and launched a "pay per crawl" marketplace that lets site owners charge AI companies for access. By the time of that announcement, Cloudflare reported that more than a million customers had already turned the AI bot block on.

The practical upshot: if you run a Cloudflare site and want a strong block, the dashboard toggle plus a complete robots.txt is more reliable than robots.txt alone. Cloudflare's block does not depend on the crawler being polite.

For servers that are not behind Cloudflare, the equivalent move is to block known AI scraper user agents at the web server level. A small Nginx snippet looks like this:

if ($http_user_agent ~* "GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot") {
  return 403;
}

This is harsher than robots.txt because it serves a 403 Forbidden response instead of asking nicely. The trade off is that user agent strings are easy to spoof, so it does not stop a determined scraper, but it does stop the well behaved ones from indexing anything at all.

what about ai.txt and llms.txt

Two newer proposals try to extend the robots.txt idea for AI specifically. They are worth knowing about, but neither is a substitute for the directives above.

ai.txt was proposed by Spawning.ai. The idea is a separate file at the root of your site that lists explicit permissions for AI training by media type and use case. Spawning operates an opt out registry that some partners, including Hugging Face and Stability AI, consult when assembling training data.

llms.txt is a different proposal aimed at inference time rather than training. It is a curated, markdown formatted summary of the site intended to be fetched by an LLM when a user asks it a question, so that the model can ground its answer in a clean version of the content. As of 2026 llms.txt is a proposal, not a standard, and no major AI company has publicly committed to consuming it as a default input. Adding one is harmless, but it does nothing to prevent training; if anything, it makes a site's content easier to use at inference.

For preventing training, robots.txt plus meta tags plus a CDN level block remain the operative tools.

protecting images: glaze and nightshade

For visual artists, robots.txt rules are not enough. Even if an image is not freshly scraped by GPTBot, copies of it may already sit in Common Crawl, LAION, or any of the many image datasets used to train Stable Diffusion and other generators. Two tools from the SAND Lab at the University of Chicago, led by professor Ben Zhao, try to address this by modifying the image itself.

Glaze was released in March 2023. It applies imperceptible pixel level perturbations to an image so that AI models reading the file learn a different style than what humans see. The intent is style cloaking: a Studio Ghibli inspired illustration looks like Ghibli to a person but trains the model as if it were, say, a Cubist painting. Glaze passed peer review with a distinguished paper award at USENIX Security 2023. By late 2024 it had been downloaded more than six million times.

Nightshade was released in late 2023 and is the offensive companion to Glaze. Instead of cloaking style, Nightshade poisons the training data: small numbers of Nightshaded images can mis associate concepts inside a generative model, so that for example prompts for "dog" begin producing cats. Nightshade was downloaded more than 1.6 million times in its first year and was accepted at the IEEE Symposium on Security and Privacy.

Neither tool prevents the image from being scraped. They prevent the scrape from being useful. They also do not undo training that has already happened on un Glazed copies.

the limits of all of this

It is worth being honest about what these techniques do and do not do.

Robots.txt is voluntary. It is a polite request, not an access control. A crawler that ignores it pays no technical cost, although it may face reputational and legal cost if it gets caught. OpenAI, Google, Anthropic, Perplexity, Apple, and Common Crawl all publicly state they respect robots.txt for their named bots, but the standard predates AI and there is no enforcement mechanism.

Old training data is not retroactively removed. Adding GPTBot to robots.txt today does nothing about content that was already in the GPT-3 or GPT-4 training set, which was assembled before GPTBot existed as an opt out. There is no public API that lets a publisher demand removal from an existing model's weights.

User agents change. New bots appear. Older ones get renamed or retired. A robots.txt written in 2023 that only listed GPTBot is now missing Google-Extended, ClaudeBot, OAI-SearchBot, Applebot-Extended, and several others. Treat the file as a living document and review it at least annually. The community maintained list at the GitHub repo ai-robots-txt/ai.robots.txt is a useful reference.

Some scrapers do not identify themselves at all. Plenty of training data is collected through unbranded scrapers, scraping as a service vendors, or bots that simply lie about their user agent. The only defense against those is a bot management service that fingerprints behavior rather than trusting the User-Agent header, which is roughly what Cloudflare's AI scraper block does.

why this matters: the legal backdrop

The rise of opt out tools is happening against a wall of copyright lawsuits. On December 27, 2023 The New York Times sued OpenAI and Microsoft in the U.S. District Court for the Southern District of New York (case 1:23-cv-11195), alleging that the companies copied millions of Times articles without permission to train ChatGPT and Bing Chat. The complaint cited examples of ChatGPT reproducing Times articles nearly verbatim and sought billions of dollars in damages plus destruction of models trained on Times content. Similar suits have been filed by Getty Images against Stability AI, by authors including Sarah Silverman and Ta Nehisi Coates against OpenAI and Meta, and by music publishers against Anthropic.

AI companies argue that training is fair use. Publishers argue that the outputs are derivative works. The cases are still being litigated as of 2026 and have not produced a single dispositive ruling, but they have already changed industry behavior: every major lab now publishes a named training bot that respects robots.txt, partly because doing so creates a paper trail of consent that helps in court.

For a website owner, the practical implication is that adding the directives in this guide does two things. It tells well behaved AI companies not to use your content, and it documents your refusal, which matters if you later need to argue that any scraping was unauthorized.

a recommended setup

If you take nothing else from this article, here is a defensible default for a typical content site in 2026:

1. Put the full robots.txt block list above at the root of your domain.
2. Add X-Robots-Tag: noai, noimageai as a default header.
3. If you are on Cloudflare, turn on the AI Scrapers and Crawlers toggle in Security and Bots.
4. If you publish original artwork, run images through Glaze before posting.
5. Review the bot list once a year and add any new training agents from the ai-robots-txt community repo.

That combination handles the major training crawlers, leaves the door open for AI search bots that link back to you, and gives you a real defense rather than just a hopeful note in a file most of the internet ignores.

references

1. OpenAI Platform. "Overview of OpenAI Crawlers." https://platform.openai.com/docs/bots
2. Google. "An update on web publisher controls." September 28, 2023. https://blog.google/technology/ai/an-update-on-web-publisher-controls/
3. Anthropic. "Does Anthropic crawl data from the web, and how can site owners block the crawler?" https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
4. Apple. "About Applebot." https://support.apple.com/en-us/119829
5. Common Crawl. "CCBot." https://commoncrawl.org/ccbot
6. Cloudflare. "Declare your AIndependence: block AI bots, scrapers and crawlers with a single click." July 3, 2024. https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
7. Cloudflare. "Control content use for AI training." https://blog.cloudflare.com/control-content-use-for-ai-training/
8. Search Engine Land. "Google introduces Google-Extended to let you block Bard, Vertex AI via robots.txt." https://searchengineland.com/google-extended-crawler-432636
9. Search Engine Journal. "Google Clarifies the 'Google-Extended' Crawler Documentation." https://www.searchenginejournal.com/google-clarifies-the-google-extended-crawler-documentation/507645/
10. Glaze Project, University of Chicago. https://glaze.cs.uchicago.edu/
11. Nightshade, University of Chicago. https://nightshade.cs.uchicago.edu/
12. MIT Technology Review. "The AI lab waging a guerrilla war over exploitative AI." November 13, 2024. https://www.technologyreview.com/2024/11/13/1106837/ai-data-posioning-nightshade-glaze-art-university-of-chicago-exploitation/
13. The Register. "Cloudflare offers 1-click block against web-scraping AI bots." July 3, 2024. https://www.theregister.com/2024/07/03/cloudflare_ai_blocks/
14. CourtListener. "The New York Times Company v. Microsoft Corporation, 1:23-cv-11195." https://www.courtlistener.com/docket/68117049/the-new-york-times-company-v-microsoft-corporation/
15. ai-robots-txt community list. https://github.com/ai-robots-txt/ai.robots.txt
16. Nieman Journalism Lab. "Cloudflare will block AI scraping by default and launches new 'Pay Per Crawl' marketplace." 2025. https://www.niemanlab.org/2025/07/cloudflare-will-block-ai-scraping-by-default-and-launches-new-pay-per-crawl-marketplace/