GPT-2

GPT-2 is a family of autoregressive language models introduced by OpenAI on February 14, 2019. The family contains four decoder-only Transformer models distributed under the names 124M, 355M, 774M, and 1.5B, referring approximately to their parameter counts. The largest model has 48 Transformer layers, a hidden width of 1,600, 25 attention heads per layer, and a context limit of 1,024 tokens. All four models use the same next-token prediction objective and were trained on WebText, a 40 GB collection derived from slightly more than 8 million web documents.^[1][2][5]

GPT-2 followed GPT-1 and retained its left-to-right generative pre-training approach, but increased both model capacity and the amount and diversity of training text. Its paper investigated whether a sufficiently capable model trained only for language modeling could exhibit task behavior from patterns present in ordinary text. The reported evaluations included language modeling, reading comprehension, summarization, translation, and question answering. Results on downstream tasks were often rudimentary or far below specialized systems, so the paper is evidence of emergent task behavior under a particular evaluation setup, not evidence that GPT-2 generally mastered those tasks.^[1][2][3]

GPT-2 also became a prominent case in debates about model release. OpenAI initially published only the smallest model, citing concern about scalable misuse of generated text, then released progressively larger checkpoints before publishing the full 1.5B model on November 5, 2019. This staged process produced model cards, detector research, an output dataset, and a technical report about publication decisions. The completed release included model code and weights, but not WebText or a complete, documented training pipeline.^{[2][5][6][7][8][9]}

Scope and terminology

"GPT-2" can mean the model family or, in some sources, only its largest member. The original paper called the 1.5-billion-parameter model GPT-2 while describing all four WebText language models as a size series. OpenAI's subsequent repository and model card apply GPT-2 to the whole family. This article uses "GPT-2 family" for all four checkpoints and "GPT-2 XL" or "1.5B model" for the largest checkpoint.^[1][5][6]

GPT-2 is a base model rather than an instruction-tuned assistant. It predicts continuations from preceding tokens and has no built-in conversation protocol, retrieval system, factuality mechanism, or alignment training comparable to later assistant models. Prompts can make some tasks resemble text completion, but the model was not trained to follow arbitrary user instructions. In the 2019 model card, OpenAI explicitly placed truth-dependent uses out of scope because the model does not distinguish fact from fiction.^[6]

The public names and the counts printed in the 2019 paper differ. The paper lists approximately 117M, 345M, 762M, and 1,542M parameters. OpenAI later stated that those counts were erroneous and distributed the checkpoints as 124M, 355M, 774M, and approximately 1.5B. The largest download directory is named 1558M. These labels refer to the same four architectures, not to additional versions.^[1][5]

Research background

GPT-1 demonstrated a two-stage method: generative language-model pre-training on unlabeled text followed by supervised fine-tuning for individual tasks. Its model had 12 Transformer layers and used task-specific input transformations during fine-tuning. GPT-2 retained autoregressive pre-training but changed the central research question. Instead of fine-tuning and measuring a different trained model for every task, its authors evaluated whether one WebText-trained model could perform tasks when those tasks were represented as text in its context.^[1][3]

The GPT-2 paper described this idea as "unsupervised multitask learning." That phrase did not mean the training set had no human influence: WebText was selected through a human-curation proxy, and the text naturally contained examples, explanations, questions, answers, and translations. It also did not mean every evaluation used a bare instruction. Some evaluations inserted a task hint, a delimiter, or example input-output pairs into the context. The central distinction was that the model's weights and architecture were not updated on the downstream benchmark's training data.^[1]

The architecture was based on the Transformer, whose original formulation replaced recurrent sequence processing with attention mechanisms. GPT-2 uses only the Transformer decoder stack, with causal masking so a token position can attend to earlier positions but not future positions. That constraint makes the network autoregressive: it estimates the probability of each next token conditional on the tokens already present.^[1][4]

Architecture

Model sizes

The four public checkpoints have the following architecture. Layer and hidden-width values come from the paper, while attention-head counts and public labels come from OpenAI's distributed hyperparameter files.^[1][5]

Scroll sideways for more →

Each attention head has a width of 64 because the hidden width divides evenly by the number of heads. For GPT-2 XL, for example, 1,600 divided by 25 is 64. The 25-head figure is therefore consistent with the published and distributed architecture; it is not a transcription error.^[1][5]

Differences from GPT-1

GPT-2 is not merely an unchanged GPT-1 network with more layers. The paper reports several architectural and training changes. Layer normalization was moved to the input of each attention or feed-forward sub-block, and another layer normalization was added after the final block. Residual-layer weights used a depth-dependent initialization scaled by the inverse square root of the number of residual layers. The vocabulary grew to 50,257 tokens, the context limit increased from 512 to 1,024 tokens, and the training batch size was 512 sequences.^[1]

The network represents token identity and position in a shared residual stream. Each layer applies masked multi-head self-attention, a position-wise feed-forward network, residual connections, and layer normalization. At generation time, the model produces logits for the next token, converts them to a probability distribution, selects or samples a token, appends it to the context, and repeats. Generation behavior therefore depends not only on model weights and prompt text but also on the decoding rule.^[1][4]

Tokenization

GPT-2 uses a byte-level form of byte-pair encoding. Text is first represented through UTF-8 bytes, then frequent byte sequences are merged into a vocabulary of 50,257 tokens. This design avoids a fixed unknown-token category and can assign a probability to any Unicode string representable as bytes. The paper also prevented merges across broad character categories, with an exception involving spaces, to reduce wasteful duplicate forms of words that differ only by surrounding punctuation or spacing.^[1]

Tokens are not the same as words. A common word may be one token, while a rare word, punctuation pattern, or non-English string may require several. Consequently, the 1,024-token limit does not map to a fixed number of words or characters. Statements that GPT-2 "predicts the next word" are a useful informal description from OpenAI's announcement, but next-token prediction is the technically precise description.^[1][2]

WebText training data

Construction

OpenAI created WebText to obtain varied web documents without starting from an unfiltered crawl. The collection process followed outbound links posted to Reddit that had received at least three karma. The paper describes 45 million candidate links. Text was extracted from the linked HTML using Dragnet and Newspaper, then deduplicated and heuristically cleaned. The preliminary corpus used for the reported experiments contained slightly more than 8 million documents and 40 GB of text.^[1]

The authors removed Wikipedia pages because Wikipedia appeared in other evaluation corpora and could make overlap analysis harder. They also excluded links created after December 2017. OpenAI's model card therefore gives the training-data cutoff as the end of 2017. The Reddit threshold was a selection heuristic intended to identify links that users had found interesting, educational, or amusing. It should not be interpreted as a validated measure of truth, representativeness, safety, or editorial quality.^[1][2][6]

WebText was not the text of Reddit discussions themselves in the paper's description. Reddit supplied the outbound URLs and the karma filter; the documents came from the linked sites. The selection process could still inherit substantial biases from Reddit's users, linked domains, language distribution, and voting behavior. Later research found toxic and factually unreliable material in WebText and in OpenWebText, an independent attempt to reproduce its collection method.^[14][15]

Availability and reproducibility

OpenAI did not publish the WebText training set. The May 2019 output-dataset release included 250,000 documents from a WebText test split, but that subset is not the complete training corpus. It also included generated examples from each model size under untruncated random sampling and top-k sampling with k equal to 40. These files enabled detector and generation research without making the original training collection reproducible.^[2][10]

The public GPT-2 repository contains inference and sampling code, checkpoint-download tooling, the model card, and model files. It does not contain the private WebText corpus or a complete training implementation with all data preparation and training details needed to recreate the original checkpoints. Describing the repository as the "original training code" is therefore inaccurate.^[2][5][6]

Training objective and optimization

For a token sequence (x_1,\ldots,x_n), GPT-2 maximizes the product of conditional next-token probabilities:

[ P(x_1,\ldots,x_n)=\prod_{i=1}^{n}P(x_i\mid x_1,\ldots,x_{i-1}). ]

This objective does not provide explicit labels for translation, question answering, or summarization. The paper's hypothesis was that diverse web documents contain textual demonstrations of many such activities and that a model with enough capacity could learn patterns that make those activities useful for next-token prediction.^[1]

OpenAI trained four approximately log-spaced model sizes and manually tuned each model's learning rate on a 5 percent held-out WebText sample. The paper states that training and held-out perplexity were still improving and characterizes all four models as underfitting WebText. This is a report about the observed training regime, not proof that continued training would indefinitely improve every downstream capability or that parameter count alone determines performance.^[1]

OpenAI's paper, repository, and model card do not disclose a complete training-compute accounting. Without a primary record or reproducible training logs, exact accelerator counts, training duration, energy use, or carbon emissions should not be treated as confirmed specifications of the original run.^[1][5][6]

Evaluation

Meaning of zero-shot

The GPT-2 paper used "zero-shot" primarily to mean that no benchmark-specific gradient updates or architectural changes were made. The exact prompting method varied by task. Language-model benchmarks were evaluated directly after reversible detokenization. Summarization used the textual cue TL;DR:. Translation was induced by placing example English-French pairs in the context. Natural Questions was also seeded with example question-answer pairs to establish the expected short-answer format.^[1]

This usage differs from later conventions that distinguish zero-shot prompts from one-shot or few-shot demonstrations. Calling every GPT-2 downstream result demonstration-free would be incorrect. The experiments are better described as no-fine-tuning evaluations, some direct and some using in-context examples or task cues.^[1]

Language-modeling benchmarks

The largest model's reported results are reproduced below. Perplexity is lower when better; accuracy is higher when better. BPB means bits per byte and BPC means bits per character. These values come from the paper's Table 3 and use its reversible detokenization procedure.^[1]

Scroll sideways for more →

The paper reported a new result on seven of eight language-modeling datasets under its comparison setup. The exception was the One Billion Word Benchmark, where the preceding result shown in the paper was 21.8 perplexity. Comparisons need care because corpora use different canonical units and preprocessing. For Penn Treebank and WikiText-2, the authors applied invertible detokenizers to reduce artifacts. The LAMBADA accuracy result also used a stop-word filter that raised accuracy from 52.66 percent to 63.24 percent, so the filtered and unfiltered numbers answer different evaluation questions.^[1]

On the Children's Book Test, one test-set book was found in WebText. The paper therefore reported validation-set results, where it found no significant overlap. More generally, the authors compared eight-gram overlap between WebText and benchmark test sets. They concluded that overlap supplied a small but consistent benefit to some reported results, while noting that common phrases and duplicates also occurred inside the benchmarks' own train-test divisions. This analysis was an early warning that web-scale pre-training makes contamination checks necessary.^[1]

LAMBADA and long-range context

LAMBADA asks a model to predict the final word of a passage when broader context is necessary. GPT-2 XL reduced perplexity from the prior 99.8 shown in the paper to 8.63. Its raw language-model accuracy was 52.66 percent; excluding stop words as candidate endings raised the reported accuracy to 63.24 percent. The paper noted that many errors were plausible continuations but invalid final words, showing that language-model probability and the benchmark's constrained answer format were not identical objectives.^[1]

These results demonstrate useful dependence on wider context within a 1,024-token window, but they do not establish unrestricted long-document reasoning. They also do not isolate whether gains came from model scale, WebText, architectural changes, optimization, or interactions among those factors.^[1]

Reading comprehension

For CoQA, GPT-2 received a document, the preceding dialogue, and a final answer marker. Greedy decoding from the largest model achieved 55 F1 on the development set without training on CoQA's more than 127,000 manually collected question-answer pairs. This matched or exceeded three of four baselines cited by the paper. The supervised BERT-based system cited in the same discussion was near 89 F1, and error inspection suggested that GPT-2 frequently used shallow retrieval heuristics such as answering a "who" question with a name from the document.^[1]

The result is therefore notable as task behavior obtained without weight updates, but it was not competitive with the strongest supervised system and does not by itself demonstrate robust reading comprehension. Around 15 percent of CoQA news-domain documents appeared in WebText, although the actual CoQA questions and answers postdated WebText's cutoff. The authors estimated a modest performance benefit from document overlap.^[1]

Summarization

The summarization experiment used CNN and Daily Mail articles. Appending TL;DR: to an article, generating 100 tokens with top-k sampling at k equal to 2, and retaining the first three generated sentences produced a ROUGE average of 21.40. Removing the textual hint reduced the average to 15.03. The specialist Bottom-Up Summarization system shown in the paper scored 32.75, while selecting three random article sentences scored 20.98.^[1]

The generated text often resembled a summary, but the paper reports that it could focus on recent material, confuse details, and only narrowly beat the random-sentence baseline under the aggregate metric. This experiment supports the narrower claim that a textual cue could elicit some summarization-like behavior from the base language model.^[1]

Translation

For translation, the model was not simply given an isolated command such as "translate English to French." The context contained example pairs formatted as an English sentence followed by its French counterpart, then a final source sentence for completion. On WMT 2014, GPT-2 XL achieved 5 BLEU for English-to-French and 11.5 BLEU for French-to-English. The paper compared the latter with 33.5 BLEU from the strongest unsupervised machine-translation approach it cited.^[1]

WebText deliberately excluded non-English pages, but a byte-level language detector found about 10 MB of French text in the corpus. The limited translation performance may therefore have drawn on multilingual fragments and naturally occurring paired examples. It did not make GPT-2 a general translation system.^[1]

Question answering

The Natural Questions evaluation seeded the context with example question-answer pairs, then asked the model to continue with short answers. GPT-2 XL answered 4.1 percent of development questions correctly by exact match, compared with 1.0 percent for the smallest model. Accuracy reached 63.1 percent within the 1 percent of questions to which the largest model assigned the highest confidence, while the paper placed contemporary retrieval-based open-domain systems in a much higher overall 30 to 50 percent range.^[1]

The confidence result shows that the model assigned useful relative probability to a small subset of answers. It does not mean the system had 63.1 percent overall question-answering accuracy. The full-development-set result remained 4.1 percent.^[1]

Winograd Schema Challenge

On the Winograd Schema Challenge, GPT-2 XL reached 70.70 percent under the paper's partial-scoring method and improved as model size increased. The paper cautioned that the dataset had only 273 examples and directed readers to work discussing the statistical uncertainty of the benchmark. The score should not be restated as conclusive evidence of human-like commonsense understanding.^[1][2]

Text generation

GPT-2 generates by repeatedly sampling or selecting a next token from its probability distribution. Greedy decoding always selects the highest-probability token. Untruncated random sampling uses the complete distribution. Top-k sampling retains only the k most probable tokens at each step, renormalizes their probabilities, and samples from that restricted set. OpenAI commonly used top-k with k equal to 40 for its 2019 samples and output dataset.^[1][10]

The original announcement displayed hand-selected long continuations. OpenAI disclosed that the examples involved selection and that obtaining a good sample usually took several attempts. The post estimated reasonable samples around half the time for topics well represented in the data and worse behavior for technical or esoteric topics. It also listed repetition, incorrect world modeling, and abrupt topic changes as recurring failure modes.^[2]

The paper's appendix provided a more controlled comparison. It used random unseen WebText contexts, 768 context tokens, and 256-token completions from the smallest and largest models with top-k 40 sampling. The larger model generally produced more coherent continuations, but the result remained conditional on the source context, decoding method, and human evaluation of examples.^[1]

Decoding changes both perceived quality and detectability. Ippolito and colleagues compared top-k, nucleus, and untruncated sampling on GPT-2 large. In their setup, expert human raters reached 71.4 percent accuracy on 192-token excerpts, while fine-tuned automatic discriminators depended strongly on excerpt length and sampling strategy. A discriminator trained on one strategy could transfer poorly to another. These results rule out a simple claim that GPT-2 text is either universally easy or universally impossible to detect.^[13]

Staged release

The release sequence can be summarized as follows. Dates refer to public announcements, and the checkpoint labels use OpenAI's corrected public names.^[2][8][9]

Scroll sideways for more →

OpenAI's August follow-up contains an internal timeline that assigns the medium-model release to April, while the same page's opening text and the updated original announcement identify it as a May release. This article uses May because that is the date stated in the actual public update and in the release report. The discrepancy is one reason to describe the intermediate event by month rather than inventing a precise day.^[2][7][8]

Initial release

On February 14, 2019, OpenAI published the paper, samples, and the smallest model with sampling code. It withheld the larger weights, WebText, and training code. The stated concern was that more capable text generation might lower the cost of spam, impersonation, abusive content, or misleading articles. OpenAI described the decision as an experiment and committed to revisiting it rather than claiming that its risk estimates were settled.^[2]

The initial policy discussion sometimes overshadowed the technical scope. The largest model was not publicly released at announcement, so it is inaccurate to call GPT-2 XL the largest publicly released language model in February 2019. Only the smallest checkpoint was public at that stage.^[2][7]

Intermediate releases

OpenAI released the 355M model in May 2019 and published the GPT-2 output dataset. The dataset included WebText test documents and generations from all four model sizes under two decoding conditions, along with detector baselines. OpenAI also shared unreleased larger models with selected research partners under agreements intended to support analysis before wider release.^[2][10]

On August 20, 2019, OpenAI released the 774M checkpoint, a technical report, and a model-sharing legal agreement. The four named research partners were Cornell University, the Middlebury Institute's Center on Terrorism, Extremism, and Counterterrorism, the University of Oregon, and the University of Texas at Austin. Their projects examined human susceptibility to synthetic disinformation, possible extremist misuse, model bias, and detectability after fine-tuning or across models.^[7][8]

The accompanying report described staged release as a way to create time for risk and benefit analysis between model sizes. It reported minimal observed evidence of misuse while acknowledging limited visibility into unobserved activity. The report did not establish that misuse was impossible; it documented the information OpenAI used in that release decision.^[7]

Full release

OpenAI released the 1.5B code and weights on November 5, 2019. The final post said that survey participants assigned mean credibility scores of 6.91 out of 10 to 1.5B outputs, 6.72 to 774M outputs, and 6.07 to 355M outputs. OpenAI interpreted the small increase from 774M to 1.5B as one factor favoring release. These figures came from a particular partner study and should not be generalized to every genre, prompt, reader population, or decoding setting.^[9]

The final post also reported about 95 percent detection performance for tested 1.5B outputs. OpenAI explicitly said that this was insufficient for a standalone detector and that results depended heavily on sampling methods. It recommended combining content-based signals with metadata, human judgment, and public education. Releasing the detector also made its behavior available to possible adversaries.^[9]

Publication-norms significance

GPT-2's release became a case study in responsible publication rather than a universal template. Partnership on AI says its publication-norms workstream was sparked by discussion of GPT-2's staged release. A later Center for Security and Emerging Technology analysis similarly treats GPT-2 as an event that accelerated debate over what to release, when to release it, and how to involve outside researchers.^[11][12]

Those sources support a historical claim about the debate, not a claim that staged release was proven optimal. Release decisions depend on model capability, plausible harms, replication costs, availability of mitigations, external review, and the costs of withholding useful research. The GPT-2 process exposed these tradeoffs and supplied an empirical case for later policy discussions.^[7][11][12]

Distribution, licensing, and implementations

OpenAI's repository distributes model code and checkpoint-download tools under a Modified MIT License. It includes the four public model configurations and the November 2019 model card. As of the research cutoff for this article, OpenAI had archived the repository on April 8, 2026, making it read-only but still downloadable. The output-dataset repository was likewise archived on May 29, 2026.^[5][10]

The repository labels are the authoritative source for the corrected checkpoint names. OpenAI's note says the parameter counts in the original paper and earlier posts were wrong, explaining why academic papers and older file names may still use 117M, 345M, 762M, or 1542M. Applications should identify both the model size and the implementation because converted checkpoints can differ in file format even when they represent the same learned weights.^[5]

Hugging Face hosts GPT-2 checkpoints, including openai-community/gpt2 for the smallest model and separate medium, large, and XL repositories. These hosted implementations make the models available through the Transformers library, but the hosting organization did not create the original GPT-2 training corpus or checkpoints. OpenAI remains the model developer.^[5][6][22]

DistilGPT2 is a separate model, not an alias for GPT-2. Its model card identifies Hugging Face as the developer, describes an 82-million-parameter model distilled from GPT-2 small, and gives it an Apache 2.0 license. It shares ancestry and a generation task with GPT-2 but has different weights, developer, and license.^[19]

Limitations and risks

Factual reliability and coherence

The model card warns that GPT-2 does not distinguish fact from fiction and discourages truth-dependent uses. OpenAI's repository also warns that the models can be subtly incoherent or inaccurate and that robustness and worst-case behavior were not well understood. These limitations follow from the training objective: next-token likelihood rewards plausible continuations, not verified truth.^[5][6]

The 1,024-token context also constrains long-range dependence. Information beyond the window is unavailable to the model unless an application separately selects or summarizes it. Even within the window, the original post documented repetition, topic switching, and incorrect physical or world descriptions. A fluent surface form is therefore not evidence that a continuation is factual or internally consistent.^[1][2]

Social bias and toxic output

WebText reflects the pages selected through its collection process, including their stereotypes, abusive language, and factual errors. The official model card says the models may reproduce training-data biases and recommends a use-case-specific bias study before deployment in systems that interact with people.^[6]

Sheng and colleagues evaluated GPT-2 generations conditioned on demographic terms. Their study found differences in positive, neutral, and negative "regard" across gender, race, and sexual-orientation prompt pairs. The work used constrained templates and a limited set of demographic categories, so it demonstrates measurable bias in that setup rather than providing a complete census of all possible social harms.^[14]

Gehman and colleagues tested GPT-2 small and other models using RealToxicityPrompts. They found that non-toxic prompts could still produce text classified as toxic, and they documented toxic or unreliable material in WebText and OpenWebText. Their measurements relied on Perspective API, whose own demographic and annotation biases the authors discuss. Both the generation result and the limits of the toxicity classifier are necessary for interpreting the study.^[15]

Memorization and privacy

The GPT-2 paper itself observed memorization of long strings that appeared repeatedly in WebText. For example, greedy continuation from the beginning of the Gettysburg Address reproduced text before drifting. Its overlap experiment found that most sampled continuations had less than 1 percent eight-gram overlap with WebText training data, but aggregate overlap does not rule out rare exact leakage.^[1]

Carlini and colleagues later mounted a black-box extraction attack against GPT-2. They generated 1,800 high-ranked candidate sequences across attack configurations and confirmed that more than 600 were verbatim training examples. Extracted material included public personal information, code, identifiers, and conversations, including examples that appeared in only one training document. This demonstrated practical extraction risk; it did not estimate the fraction of all WebText documents that could be recovered.^[16]

The study discussed deduplication and differential privacy as possible mitigations, while noting tradeoffs and that empirical safeguards do not provide the same guarantee as properly configured private training. GPT-2 was not trained with a stated differential-privacy guarantee.^[16]

Synthetic-text detection

No single accuracy value describes GPT-2 detection. OpenAI's approximately 95 percent figure applied to its tested 1.5B generation setup and was rejected by OpenAI as adequate standalone performance. The output-dataset repository reported mid-90s accuracy for top-k 40 samples but lower results for untruncated random samples, plus evidence that fine-tuning could reduce detectability.^[9][10]

Peer-reviewed work further showed sensitivity to sample length, decoding rule, conditioning, training distribution, and transfer across strategies. Human and automatic detectors relied on different cues. A detector can therefore fail when the generation process or domain differs from its training data, and a human reader can mistake fluent output for human writing. Claims that GPT-2 created a solved detector problem, or an inherently undetectable one, are both unsupported.^[13]

Data transparency

Because WebText was never fully released, external researchers cannot audit every source document, reconstruct all removals, or reproduce the exact data pipeline from public artifacts. The 250,000-document test subset supports partial study but does not replace the training corpus. This limits independent analysis of provenance, consent, copyright, demographic representation, and benchmark contamination.^[1][6][10]

Research influence

Scaling and later GPT models

GPT-2 supplied an early controlled size series in which many measured behaviors improved with model capacity. The paper plotted performance across four models and noted that even GPT-2 XL underfit WebText. These observations helped motivate continued study of model and data scaling, but the experiment did not isolate "scale alone": the comparison existed within one architecture family, objective, corpus, and optimization setup.^[1]

GPT-3 continued the autoregressive Transformer line at much larger scale and made in-context few-shot evaluation a central method. Its paper explicitly cites GPT-2 among the preceding generative pre-training systems. This establishes a technical lineage, while avoiding the stronger and unsupported claim that any single GPT-2 result uniquely determined GPT-3's design.^[20]

Datasets and replications

WebText's non-release motivated public approximations. OpenWebText was created independently by following a similar Reddit-link collection idea. RoBERTa used a 38 GB OpenWebText corpus and explicitly described it as an open-source recreation of WebText; its authors also stated that they were not affiliated with OpenWebText's creation. OpenWebText is therefore related training data, not a public copy of OpenAI's private corpus.^[17][21]

The distinction matters for reproducibility. A result obtained with OpenWebText can test a similar data recipe, but differences in crawl date, reachable pages, extraction software, filtering, deduplication, and removed domains can change the corpus. It cannot establish the exact contents of WebText.^[1][17][21]

Interpretability research

GPT-2 small became a common object of mechanistic interpretability research because its weights and architecture are public and small enough for controlled interventions. Wang and colleagues analyzed an indirect-object-identification task using synthetic templates. They proposed a circuit of 26 attention heads grouped into seven functional classes and evaluated it using path patching, knockouts, faithfulness, completeness, and minimality tests.^[18]

The study is evidence that a specific behavior in GPT-2 small can be partially reverse-engineered. It is not a complete explanation of GPT-2, natural language understanding, or even every instance of indirect-object use. The authors reported remaining gaps and found redundant or backup behavior that complicated causal explanations.^[18]

Model cards and release artifacts

GPT-2's November 2019 model card recorded model date, intended research uses, unsupported truth-dependent uses, and bias warnings. Alongside the staged-release report and output dataset, it provided documentation beyond a paper and weight files. These artifacts became part of the historical discussion about documenting model limitations and structuring external research around a release.^{[6][7][10][11]}

The output dataset deserves separate treatment because it is often confused with WebText. For each model size, OpenAI published a training split of 250,000 generated examples plus validation and test splits of 5,000 examples. It supplied both untruncated temperature-1 sampling and top-k 40 sampling, as well as 250,000 WebText test documents. File names retained the paper's older labels, such as small-117M and large-762M, even after the repository corrected the checkpoint counts. The dataset also included samples from a full model fine-tuned on Amazon reviews.^[10]

These artifacts supported three different kinds of study. The WebText test subset provided human text from a distribution close to the model's training data. The paired generation conditions allowed researchers to measure how decoding changes detector accuracy. The fine-tuned samples exposed the risk that a detector trained only on base-model output would not generalize to a modified generator. None of the three artifacts disclosed the complete WebText training set.^[10][13]

Interpreting historical claims

GPT-2 is sometimes described retrospectively using terms that became standard only after 2019. It is reasonable to call the family a large language model in the historical lineage of modern LLMs, but its largest 1.5B checkpoint is small by later parameter scales. Size labels are relative to the period and family. "Large" in gpt2-large identifies the 774M member, not a timeless capability class.^[1][5]

Likewise, "zero-shot" in the paper should be read according to the experimental procedure, not only according to later prompt-taxonomy conventions. Translation and Natural Questions used examples in context, which later literature would usually call few-shot prompting. Summarization used a task cue, while language-model transfer involved direct scoring. The shared property was no task-specific weight update.^[1]

The paper title is also a research claim, not a settled definition of GPT-2. The experiments showed that a next-token model could express several task-like behaviors from context. They did not show that the model had learned every task in WebText, that its behavior was reliable outside the measured prompts, or that language modeling eliminated the need for supervised data. Several reported downstream results were explicitly far below specialist systems.^[1][2]

See also

• Natural Language Processing
• AI safety
• Common Crawl

References

1. ^Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, "Language Models are Unsupervised Multitask Learners," OpenAI, 2019. cdn.openai.com/...language-models.pdf
2. ^Alec Radford et al., "Better language models and their implications," OpenAI, February 14, 2019, with May 2019 update. openai.com/...better-language-models
3. ^Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, "Improving Language Understanding by Generative Pre-Training," OpenAI, 2018. cdn.openai.com/...language_understanding_paper.pdf
4. ^Ashish Vaswani et al., "Attention Is All You Need," Advances in Neural Information Processing Systems 30, 2017. arxiv.org/...1706.03762
5. ^OpenAI, "gpt-2: Code and models from the paper 'Language Models are Unsupervised Multitask Learners'," GitHub repository and archive. github.com/...gpt-2
6. ^OpenAI, "GPT-2 model card," last updated November 2019. github.com/...model_card.md
7. ^Irene Solaiman et al., "Release Strategies and the Social Impacts of Language Models," OpenAI, 2019. cdn.openai.com/GPT_2_August_Report.pdf
8. ^OpenAI, "GPT-2: 6-month follow-up," August 20, 2019. openai.com/...gpt-2-6-month-follow-up
9. ^OpenAI, "GPT-2: 1.5B release," November 5, 2019. openai.com/...gpt-2-1-5b-release
10. ^OpenAI, "gpt-2-output-dataset," GitHub repository and archive. github.com/...gpt-2-output-dataset
11. ^Partnership on AI, "Publication Norms for Responsible AI." partnershiponai.org/...on-norms-for-responsible-ai
12. ^Helen Toner, "GPT-2 Kickstarted the Conversation About Publication Norms in the AI Research Community," Center for Security and Emerging Technology, May 1, 2020. cset.georgetown.edu/...n-the-ai-research-community
13. ^Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck, "Automatic Detection of Generated Text is Easiest when Humans are Fooled," Proceedings of ACL 2020, pages 1808-1822. aclanthology.org/2020.acl-main.164
14. ^Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng, "The Woman Worked as a Babysitter: On Biases in Language Generation," Proceedings of EMNLP-IJCNLP 2019, pages 3407-3412. aclanthology.org/D19-1339
15. ^Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith, "RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models," Findings of EMNLP 2020, pages 3356-3369. aclanthology.org/2020.findings-emnlp.301
16. ^Nicholas Carlini et al., "Extracting Training Data from Large Language Models," 30th USENIX Security Symposium, 2021, pages 2633-2650. usenix.org/...carlini-extracting
17. ^Yinhan Liu et al., "RoBERTa: A Robustly Optimized BERT Pretraining Approach," 2019. arxiv.org/...1907.11692
18. ^Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt, "Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small," International Conference on Learning Representations, 2023. arxiv.org/...2211.00593
19. ^Hugging Face, "DistilGPT2 model card." huggingface.co/...distilgpt2
20. ^Tom B. Brown et al., "Language Models are Few-Shot Learners," Advances in Neural Information Processing Systems 33, 2020. arxiv.org/...2005.14165
21. ^Aaron Gokaslan and Vanya Cohen, "OpenWebText Corpus." skylion007.github.io/OpenWebTextCorpus
22. ^Hugging Face, "GPT-2 model card and hosted checkpoints." huggingface.co/...gpt2