DistilGPT2

DistilGPT2 is an English-language autoregressive language model released by Hugging Face on October 3, 2019. It is a six-layer GPT-2-style student model pretrained under the supervision of OpenAI's smallest GPT-2 checkpoint. Hugging Face describes DistilGPT2 as an approximately 82-million-parameter model intended to retain much of GPT-2 small's text-generation behavior with fewer parameters and lower inference cost.^[1][2]

DistilGPT2 is a separate model, not another name or redirect for GPT-2. It has its own learned weights, repository, developer, training corpus, and Apache 2.0 license. The student has six Transformer layers, while GPT-2 small has twelve. DistilGPT2 was trained on OpenWebText, an independent public reproduction of the collection method used for OpenAI's unreleased WebText corpus; GPT-2 small was trained by OpenAI on WebText itself.^[1][2][8][9]

Name and release

The name is short for "Distilled-GPT2." Hugging Face announced the weights on October 3, 2019, alongside an update about the DistilBERT paper. The historical release page says that the same broad distillation method was applied to GPT-2, but the paper itself is specifically about DistilBERT. Its reported 40 percent size reduction, 60 percent speed increase, and 97 percent retained language-understanding performance are DistilBERT results and should not be attributed to DistilGPT2.^[2][7]

The DistilGPT2 release page separately described the student as 33 percent smaller and, on average, twice as fast as GPT-2. Those are developer-reported summary claims. The page does not specify hardware, software versions, batch size, sequence length, or a latency distribution for the speed comparison, so "twice as fast" is not a hardware-independent performance guarantee.^[2]

Architecture

DistilGPT2 uses the GPT-2 decoder-only Transformer architecture with causal self-attention. Its configuration declares a vocabulary of 50,257 tokens, a hidden width of 768, 12 attention heads, six Transformer layers, and 1,024 positions. The model card says that it uses the same byte-level byte-pair encoding tokenizer as GPT-2.^[1][3]

Scroll sideways for more →

The architecture values come from the distributed configuration files rather than being inferred from the model names, and the license labels come from the two repositories' own metadata or license file. OpenAI's original paper printed 117M for the smallest GPT-2 architecture, while its released checkpoint is commonly labeled 124M; OpenAI later documented that the earlier paper counts were incorrect. That historical naming issue does not create a second teacher model.^{[1][3][9][10][12][13]}

Parameter-count conventions

The original DistilGPT2 documentation rounds the model to 82 million parameters. Counting the learned parameter tensors represented by the pinned configuration and checkpoint gives 81,912,576 parameters, which rounds to that label.^[1][2][3]

Current Hugging Face metadata also displays 88,204,032 float32 tensor elements for the safetensors artifact. The extra 6,291,456 elements are six serialized 1 x 1 x 1,024 x 1,024 causal-attention masks, one per layer. The applicable Transformers implementation registers these masks as buffers, not learned parameters. The 82M and 88.2M figures therefore count different things; the larger figure does not indicate that the architecture gained trainable layers or weights.^[4][5]

Distillation and training

Knowledge distillation trains a smaller student model to reproduce behavior learned by a larger teacher. A common form uses the teacher's full output-probability distribution as a soft training target, sometimes at an elevated softmax temperature, so that the student receives information beyond the single observed next token.^[6]

Hugging Face says that DistilGPT2 was pretrained with supervision from GPT-2 small and followed a procedure similar to DistilBERT's. The DistilBERT paper combines a language-modeling loss, a distillation loss, and a cosine-distance loss, and initializes the student from alternating teacher layers. The public DistilGPT2 sources establish the teacher-student relationship and reduced layer count, but they do not provide a complete checkpoint-specific training command and log. Exact DistilBERT loss weights or ablation results should therefore not be presented as independently reported DistilGPT2 hyperparameters or results.^[1][2][7]

Training data

DistilGPT2 was trained on OpenWebText, not on OpenAI's private WebText corpus. OpenWebText was created by Aaron Gokaslan and Vanya Cohen as an open-source effort to reproduce WebText's web-document collection approach. Its project page says that URLs were taken from Reddit submissions, non-HTML and non-English pages were filtered, near-duplicate documents were removed, and documents shorter than 128 tokens were excluded. The resulting release contained 8,013,769 documents and 38 GB of text, or 40 GB under decimal units.^[1][8]

Calling OpenWebText a copy of WebText would be inaccurate. It is an independent reconstruction with different collection, availability, filtering, and extraction details. DistilGPT2 and GPT-2 small therefore share a model lineage and tokenizer, but they were not pretrained on an identical published corpus.^[8][9]

The model card estimates 149.2 kg of carbon-dioxide-equivalent emissions for training, based on eight 16 GB V100 GPUs running for 168 hours. It also states that the compute region was unavailable and East US was assumed for the calculation. This is a model-card estimate based partly on an assumed location, not a direct measurement of the training run's emissions.^[1]

Evaluation

The current model card reports a WikiText-103 test-set perplexity of 21.1 for DistilGPT2 and 16.3 for GPT-2 after each was fine-tuned on the benchmark's training set. Lower perplexity is better. Hugging Face's revision-pinned model metadata marks the 21.1 result as unverified.^[1][4]

An older Transformers v2.2.0 release page instead reports 18.5 for DistilGPT2 and 15.0 for GPT-2 under a similarly worded WikiText-103 fine-tuning comparison. The official sources do not explain the change in values or provide enough evaluation detail to reconcile it. The two pairs should be treated as versioned, developer-reported results rather than combined into one definitive comparison. In both versions, DistilGPT2 has higher perplexity than its larger teacher under the stated setup.^[1][2]

These comparisons measure predictive performance after benchmark-specific fine-tuning. They do not establish a universal quality ratio for generated text, and they do not isolate how much of any difference comes from parameter count, distillation, training data, fine-tuning, or evaluation implementation.

Bias evaluation

The model card warns that DistilGPT2 can reproduce social biases inherited from its training data and teacher behavior. It includes generated examples to demonstrate the issue but explicitly says that they are not a comprehensive stress test.^[1]

A peer-reviewed Findings of ACL 2022 study evaluated the public Hugging Face DistilGPT2 checkpoint and GPT-2 small with BOLD profession prompts. Under the study's gender-polarity metric, averaged over five evaluation runs, DistilGPT2 had an average equitability ratio of 0.508 compared with 0.561 for GPT-2 small; its minimum ratio across profession groups was 0.199 compared with 0.311. The authors interpreted the lower values as greater gender disparity for the distilled checkpoint in that experiment.^[11]

That result has a defined scope. The metric used binary male and female categories, selected profession prompts, debiased word embeddings, and a polarity threshold. The paper itself notes limitations in the metric and in binary-gender analysis. It supports a claim about measured occupational gender disparity under that setup, not a claim that every DistilGPT2 output is more biased than every GPT-2 output.^[11]

Use and distribution

The canonical Hugging Face repository is distilbert/distilgpt2. Its configuration uses the GPT2LMHeadModel architecture, and the Hub identifies it for causal text generation with the Transformers library. The repository contains weights and tokenizer artifacts for several supported frameworks and distributes the model under Apache 2.0.^[1][3][4]

DistilGPT2 can continue a prompt, generate text, or serve as a starting checkpoint for task-specific fine-tuning. It is a base causal language model, not an instruction-tuned assistant. It has no built-in retrieval system, source verification, conversation protocol, or mechanism that distinguishes a factual statement from a plausible continuation.^[1][12]

Because it shares GPT-2's 1,024-position configuration, input beyond that configured context cannot be processed as one uninterrupted sequence without an application-level strategy such as truncation or chunking. The token limit is not a word limit: byte-level BPE can represent one word with one token or split it across several tokens.^[1][3][9]

Limitations

DistilGPT2 inherits important limitations from GPT-2-style causal language modeling and from web-scale training data:

• Generated text can be fluent but false, internally inconsistent, repetitive, or unrelated to the prompt. It should not be treated as a verified information source.^[1][12]
• OpenWebText can contain stereotypes, abusive language, errors, and other undesirable material from the web. Distillation does not by itself remove those properties.^[1][8][11]
• The model card identifies English as the supported language. Byte-level tokenization can encode other languages, but encodability is not evidence of equal training coverage or evaluated quality.^[1][3]
• Generation behavior depends on the prompt and decoding settings as well as the checkpoint. A single hand-selected completion cannot characterize worst-case or typical behavior.^[1]
• Smaller parameter count does not guarantee a fixed latency, memory, or energy advantage on every device. Runtime depends on framework, numerical precision, sequence length, batch size, hardware, and implementation.^[2][4]

The model card recommends use-case-specific evaluation, especially for applications involving people or requiring factual accuracy. OpenAI's GPT-2 model card likewise places truth-dependent uses out of scope because GPT-2-style models do not distinguish fact from fiction.^[1][12]

See also

• GPT-2
• Knowledge Distillation
• Autoregressive Model
• Text Generation Models

References

1. ^Hugging Face. "DistilGPT2" model card, repository revision `2290a62682d06624634c1f46a6ad5be0f47f38aa`. huggingface.co/...README.md
2. ^Hugging Face. "Distil*" training and release documentation, Transformers v2.2.0. github.com/...README.md
3. ^Hugging Face. DistilGPT2 `config.json`, repository revision `2290a62682d06624634c1f46a6ad5be0f47f38aa`. huggingface.co/...config.json
4. ^Hugging Face. DistilGPT2 model metadata, repository revision `2290a62682d06624634c1f46a6ad5be0f47f38aa`. huggingface.co/...82d06624634c1f46a6ad5be0f47f38aa
5. ^Hugging Face. `modeling_gpt2.py`, Transformers v4.0.0. github.com/...modeling_gpt2.py
6. ^Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. "Distilling the Knowledge in a Neural Network." 2015. arxiv.org/...1503.02531
7. ^Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." EMC2 at NeurIPS 2019. arxiv.org/...1910.01108
8. ^Aaron Gokaslan and Vanya Cohen. "OpenWebText Corpus." 2019. skylion007.github.io/OpenWebTextCorpus
9. ^Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. "Language Models are Unsupervised Multitask Learners." OpenAI, 2019. cdn.openai.com/...language-models.pdf
10. ^OpenAI. GPT-2 small `hparams.json`. openaipublic.blob.core.windows.net/...hparams.json
11. ^Umang Gupta et al. "Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal." Findings of ACL 2022, pp. 658-678. aclanthology.org/2022.findings-acl.55
12. ^OpenAI. "GPT-2 Model Card," repository revision `9b63575ef42771a015060c964af2c3da4cf7c8ab`. github.com/...model_card.md
13. ^OpenAI. GPT-2 `LICENSE`, repository revision `9b63575ef42771a015060c964af2c3da4cf7c8ab`. github.com/...LICENSE