Language Model

A language model is a model of patterns in language that assigns probabilities or comparable scores to linguistic sequences. Its units may be characters, words, subwords, bytes, or other tokens. A generative language model can estimate how probable a complete sequence is, predict a continuation from preceding context, or sample a new sequence. Models trained with masked or denoising objectives are also commonly called language models, although their scores do not always define the same kind of normalized left-to-right probability distribution.^[5][13][16]

Language modeling is a method within natural language processing, not a synonym for the whole field. A large language model is a large, broadly trained neural member of this model class. A foundation model is defined by broad pretraining and adaptability and may operate on language, images, audio, or other data. Generative AI is broader still, covering systems that create content across multiple media. These terms overlap, but they describe different properties.^[15][37]

Language-model probabilities are learned from data and reflect the model, corpus, representation, and training objective. They are not certificates of grammar, meaning, factual truth, or human intent. A fluent continuation can be false, harmful, or inappropriate, while a rare but accurate sentence can receive a relatively low probability. For that reason, evaluation must connect a model and metric to a specified use case rather than treat one likelihood or benchmark number as a complete measure of language ability.^[28][35][36]

Definition and Scope

Sequence Probability

For a token sequence x_1, x_2, ..., x_T, the probability chain rule gives:

P(x_1, ..., x_T) =
  P(x_1) P(x_2 | x_1) ... P(x_T | x_1, ..., x_(T-1))

An autoregressive language model estimates each conditional factor. In practice, a beginning-of-sequence symbol can establish the initial context, and an end-of-sequence symbol allows the model to assign probability to the sequence stopping at a particular point. The same factorization applies whether the conditional probabilities come from count tables, a feedforward network, a recurrent network, or a transformer decoder.^[5][7][10]

The chain rule itself is exact. The approximation lies in how the conditional factors are estimated. An n-gram model discards all but a fixed number of recent tokens. A recurrent model compresses its processed history into a state vector. A transformer attends over positions available within its context and attention mask. Each design places different limits on what information can influence a prediction.

A conditional language model also receives an additional input c, such as a source sentence, document, dialogue history, or task instruction:

P(y_1, ..., y_T | c) =
  product over t of P(y_t | c, y_1, ..., y_(t-1))

This formulation covers many sequence-to-sequence systems. It separates the conditioning information from the output sequence while retaining the same autoregressive factorization for the output. A model can therefore be a language model even when its purpose is not unrestricted text continuation.

Learning the Distribution

Training data provides examples of sequences, not the true distribution of a language. In maximum likelihood estimation, parameters are chosen to increase the probability assigned to observed training sequences. For an autoregressive neural model, this is usually implemented by minimizing the negative log probability of the observed next token at every predicted position. The average negative log likelihood is a cross-entropy estimate on the sampled corpus.^[5][7]

This objective rewards accurate prediction under the training distribution. It does not explicitly require a model to verify a statement, follow an instruction, represent a speaker's intention, or distinguish a quotation from an endorsement. Those behaviors may be influenced by data curation, architecture, later adaptation, retrieval, or system-level controls, but they are not logical consequences of likelihood training.

The event space also depends on representation. A word-level model and a byte-level model assign probabilities to different sequences of events even when both encode the same visible sentence. Vocabulary construction, Unicode normalization, whitespace handling, special tokens, and segmentation rules are therefore part of the mathematical model, not merely an input convenience.^[5][20]

Generative Models and Representation Models

In its strict probabilistic sense, a language model supplies a distribution or score over possible linguistic sequences. In modern usage, the term also covers models pretrained by predicting hidden or corrupted text. A masked model predicts a token from both left and right context. A denoising encoder-decoder reconstructs clean text from a corrupted input. These objectives can produce useful contextual representations without giving the same direct sequence probability as a causal model.^[13][14][16]

The distinction matters when interpreting outputs. A causal model directly supplies conditional next-token probabilities and can generate from left to right. A masked model can score positions through repeated masking or pseudo-likelihood, but that score should not be presented as ordinary causal likelihood. An encoder representation extracted from a language-model objective can support classification or retrieval without generating text at all.

Historical Development

Information Theory and Statistical Modeling

Claude Shannon's 1948 information-theory paper represented discrete information sources as stochastic processes and illustrated increasingly constrained approximations to English using letter and word statistics. His 1951 study estimated the predictability and redundancy of printed English through human next-character prediction experiments. These papers established entropy and conditional prediction as tools for studying language, but they did not prescribe today's neural architectures or claim that prediction alone captured meaning.^[1][2]

Statistical language modeling became a practical component of speech systems as larger digital corpora and computing resources became available. In 1983, Lalit Bahl, Frederick Jelinek, and Robert Mercer described continuous speech recognition as maximum-likelihood decoding with statistical models of acoustic and linguistic processes. A language model supplied a prior over candidate word sequences, helping a decoder choose among acoustically plausible transcriptions.^[3]

The same probabilistic decomposition became important in early statistical machine translation. The IBM approach described by Peter Brown and colleagues in 1990 combined a target-language model with a translation model and searched for a sentence maximizing their product. The language component favored plausible target-language sequences, while the translation component represented the relationship to the source.^[4]

These systems made the sparse-data problem central. A corpus contains only a small fraction of possible word sequences. Count-based estimators therefore needed ways to allocate probability to unseen events, combine multiple context lengths, and make efficient use of limited observations. Smoothing and backoff methods became core language-model engineering techniques.^[5][6]

Neural Language Models

Neural approaches replaced discrete identities with learned continuous representations. In 2003, Yoshua Bengio and colleagues presented a feedforward neural probabilistic language model that learned a distributed vector for each word jointly with a conditional probability function. Similar words could acquire nearby representations, allowing evidence from one observed sequence to affect predictions for related sequences. The model still used a fixed-length context, but it reduced dependence on exact n-gram matches.^[7]

The word embedding idea did not make sparsity disappear. Rare tokens, limited context, expensive output normalization, and optimization remained problems. It changed the form of generalization: parameters were shared through continuous representations and network layers rather than only through count aggregation.

Recurrent neural networks offered a variable-length computational history by updating a hidden state at each position. Standard recurrent training can suffer from gradients that shrink or grow over many time steps. The Long Short-Term Memory (LSTM) architecture introduced gated memory mechanisms to improve learning over long intervals. Its original 1997 experiments concerned artificial long-lag tasks, not a claim that an LSTM perfectly retains arbitrary text histories.^[8]

In 2010, Tomas Mikolov and colleagues applied a simple recurrent network to language modeling and reported improvements over backoff models in their speech-recognition experiments. The result was evidence from particular corpora and model combinations, not a universal percentage advantage for every recurrent model. Recurrent computation still processed sequence positions in order, limiting parallelism during training.^[9]

Transformers and Pretraining

The 2017 transformer architecture replaced recurrent sequence processing with attention and position-dependent representations in an encoder-decoder translation system. Its self-attention layers let each position combine information from other permitted positions, while positional encodings represented order. The original paper reported better translation results and more parallelizable training than the recurrent and convolutional baselines it tested. It did not establish that every transformer is a language model or that attention removes all context and computational limits.^[10]

Transformers became a common architecture for several different language-model objectives:

Scroll sideways for more →

In 2018, ELMo used internal states from a pretrained bidirectional LSTM language model as contextual representations for downstream tasks. OpenAI's first GPT report combined a transformer decoder, generative pretraining, and supervised fine-tuning. BERT then pretrained a transformer encoder with masked-token prediction and a next-sentence task, using both left and right context within each layer.^[11][12][13]

Encoder-decoder pretraining extended this line. BART corrupted text and trained a model to reconstruct the original sequence, while T5 compared transfer-learning choices in a unified text-to-text framework and used a span-corruption objective in its main setup. These systems show why architecture, attention mask, corruption process, and objective must be described separately.^[14][15]

Count-Based Language Models

N-Gram Estimation

An n-gram model assumes that the next token depends only on the preceding n - 1 tokens. A bigram uses one preceding token; a trigram uses two. With corpus counts C, the maximum-likelihood trigram estimate is:

P(w_t | w_(t-2), w_(t-1)) =
  C(w_(t-2), w_(t-1), w_t) /
  C(w_(t-2), w_(t-1))

This estimator is transparent and efficient, but an unseen trigram receives probability zero. Because a sequence probability is a product, one zero factor makes the entire sequence probability zero. Increasing n captures a longer local context but makes counts sparser and tables larger.^[5][6]

Word n-grams were widely used in speech recognition, spelling correction, optical character recognition, and statistical translation. Character n-grams remain useful when word boundaries are unavailable or when robustness to spelling and morphology matters. The appropriate unit and order depend on the language, corpus size, latency constraints, and application.^[5]

Smoothing, Backoff, and Interpolation

Smoothing moves probability mass away from observed events so that unseen events receive nonzero probability. Simple additive smoothing is easy to explain but can allocate too much mass to the enormous set of unseen word n-grams. More effective methods use count-of-counts information, discount observed counts, or condition lower-order estimates on how broadly a token appears across contexts.^[5][6]

Backoff and interpolation combine context lengths in different ways. A backoff model uses a shorter history when the longer event lacks adequate evidence, with weights chosen so the distribution remains normalized. An interpolated model combines higher- and lower-order probabilities even when the longer event was observed. Katz backoff, Jelinek-Mercer interpolation, and Kneser-Ney smoothing are related but not interchangeable procedures.

Kneser-Ney smoothing is notable because its lower-order distribution reflects continuation diversity rather than raw unigram frequency alone. A word that follows many different histories receives a different lower-order role from one that is frequent mainly inside a small number of fixed phrases. Chen and Goodman's large empirical comparison found that smoothing performance varied with corpus, training size, n-gram order, and implementation details; modified Kneser-Ney methods were strong across many of their tested conditions.^[6]

Count models still have practical advantages. Their estimates can be inspected, updated from counts, compressed into specialized data structures, and combined with finite-state decoders. They can be appropriate when data, memory, or latency is limited. Their main restriction is not that they are "non-AI," but that their generalization is tied to engineered equivalence classes and short or otherwise structured histories.

Neural Architectures

Feedforward Networks

A feedforward neural network language model maps a fixed number of context tokens to embeddings, combines them through hidden layers, and predicts a distribution over the next token. Sharing embedding and hidden-layer parameters lets the model generalize across related contexts. The fixed input window still prevents a distant token from directly influencing the prediction unless it is summarized through additional features.^[7]

Computing a normalized probability over a large vocabulary can be expensive. Research has used hierarchical outputs, sampled losses, noise-contrastive methods, adaptive softmax layers, and vocabulary restrictions to reduce training or inference cost. These approximations affect what is trained or how probabilities are computed, so reported likelihoods require the exact normalization method.

Recurrent Networks

A recurrent model updates a hidden state from the current token and previous state. Unlike a fixed n-gram window, it can in principle carry information from earlier positions. In practice, the state has finite capacity, training is approximate, gradients can be unstable, and useful memory declines with task and architecture. Gated recurrent units and LSTMs were designed to improve information flow, not to provide lossless memory.^[8][9]

Recurrent models support left-to-right generation and can process streams without storing a full attention matrix. Their sequential dependency makes training less parallel than a transformer over the same sequence. This tradeoff helps explain architectural change, but it does not imply that recurrent models stopped being valid or useful.

State-space models

State-space sequence models process a sequence by updating a finite-dimensional latent state and mapping that state to an output. Structured parameterizations can support recurrent evaluation for streaming or generation and parallel convolutional computation during training. Their time and memory scaling can be linear or near-linear in sequence length, but recurrent evaluation carries earlier information through the current state rather than an explicit cache of every preceding token representation.^[38]

Selective state-space models make parts of the state update depend on the current input. Mamba combined that selection mechanism with a hardware-aware recurrent scan and reported causal language-model results competitive with the transformer baselines tested at comparable sizes. Those experiments showed that an attention-free architecture could be viable for language modeling; they did not establish a universal advantage over transformers at every model size, task, or context length.^[38]

Architecture and training objective remain separate. A state-space backbone can use next-token prediction, while a transformer can use causal, masked, denoising, or diffusion objectives. Hybrid systems can also mix attention with recurrent, convolutional, or state-space layers, so the architecture name alone does not identify a model's probability factorization.^{[10][13][14][38]}

Transformer Models

A transformer represents each position through attention-weighted combinations of other positions and position-wise transformations. In a causal decoder, an attention mask prevents a position from using future output tokens during next-token training. In a bidirectional encoder, positions may attend to visible context on both sides. An encoder-decoder adds cross-attention from output positions to an encoded input.^[10]

Transformer training can process positions in parallel once an input sequence is available. Autoregressive generation remains sequential at the token level because each new conditional distribution depends on generated predecessors. Attention also has a context window and computational cost; it is not access to all text ever seen by the model.

Transformers describe an architecture family. "Autoregressive," "masked," and "denoising" describe training or factorization choices. Conflating them causes errors such as treating BERT probabilities as GPT probabilities or assuming that every model with self-attention can generate text left to right without an added decoder.

Training Objectives

Autoregressive Prediction

A causal language model predicts each token from permitted earlier tokens. Teacher forcing supplies the actual earlier training tokens when computing losses. At inference time, generated tokens become part of later context, so early mistakes can change the distribution of everything that follows.

Autoregressive likelihood is normalized one step at a time and therefore gives a direct probability to a tokenized sequence, subject to the model's vocabulary and stopping convention. It supports continuation, conditional generation, likelihood ranking, and compression-style evaluation. A next-token objective can produce representations useful for many tasks, but this empirical transfer does not redefine the loss as a direct objective for truth or reasoning.^[7][12]

Masked Prediction

A masked language model hides selected input tokens and predicts them from the remaining visible text. BERT's setup used a transformer encoder so predictions could condition on context on both sides. Because only selected tokens are reconstructed and the input is corrupted during scoring, an MLM does not directly supply the same left-to-right joint probability as a causal model.^[13]

One way to score a complete sentence is to mask each position in turn and sum the conditional log probabilities. Salazar and colleagues called this a pseudo-log-likelihood and derived pseudo-perplexity from it. Those values can be useful for comparison and rescoring, but they should be labeled as pseudo-likelihood quantities rather than silently mixed with causal perplexities.^[16]

Denoising and Sequence-to-Sequence Objectives

Denoising models transform a corrupted sequence back into a clean one. Corruptions can mask spans, delete tokens, permute sentences, or replace spans with sentinel symbols. BART compared several noise functions and used a bidirectional encoder with a left-to-right decoder. T5 used text-to-text tasks and span corruption, predicting missing spans as a target sequence.^[14][15]

These objectives train conditional distributions of an output given a corrupted input. They are suitable for tasks where an input is transformed into another sequence, but reconstruction success is not identical to an unconditional model of natural text. Objective names should therefore accompany model scores and architecture labels.

Diffusion language modeling

Diffusion language models define a forward process that progressively corrupts text and learn a reverse process that reconstructs or generates a sequence through multiple refinement steps. For discrete tokens, the corruption can use transition matrices or an absorbing mask state. This differs from masking a selected set of positions once for representation learning because diffusion training covers multiple noise levels and specifies a reverse generative process.^[39][40]

Masked diffusion objectives can still be closely related to masked language modeling. Sahoo and colleagues derived an objective expressed as a weighted mixture of masked cross-entropy losses and connected it to a variational lower bound. Their models used encoder-only networks and supported fixed-length as well as semi-autoregressive variable-length sampling. A reverse process can update several positions between denoising steps rather than commit to one left-to-right order.^[40]

Likelihood, latency, and output quality depend on the corruption schedule, refinement-step count, sampler, and sequence-length treatment. Diffusion and causal perplexities should not be compared without compatible likelihood definitions, token units, and evaluation data. In the authors' NeurIPS 2024 experiments, masked diffusion approached but did not generally surpass the selected autoregressive perplexity baselines; that result is specific to the reported models and settings.^[5][29][40]

Tokenization and Representation

Choosing Linguistic Units

Tokenization defines the units whose probabilities a model predicts. Word-level vocabularies give short sequences for common words but require a policy for unseen forms. Character and byte models can use small base vocabularies and reduce unknown-token problems, but they create longer sequences. Subword systems occupy a middle ground by representing frequent strings as single tokens and less frequent strings as combinations.^[17][20]

Token boundaries are computational choices, not universal linguistic facts. Written languages differ in whether and how they mark word boundaries. A token can be a whole word in one context, part of a word in another, whitespace joined to a following string, a byte, or a reserved control symbol. The visible string alone is insufficient to infer the token sequence without knowing the tokenizer version and normalization rules.

Byte-Pair Encoding was adapted to open-vocabulary neural translation by repeatedly merging frequent adjacent symbols in the training data. Sennrich and colleagues showed that subword sequences could represent rare words without a separate dictionary backoff in their translation experiments. The resulting pieces need not correspond to morphemes, and the learned merges depend on the corpus and vocabulary budget.^[17]

The unigram segmentation model begins with a candidate subword vocabulary and removes pieces while optimizing a probabilistic segmentation objective. Kudo also proposed sampling among possible segmentations during training as subword regularization. This is distinct from BPE even though both produce subword sequences.^[18]

SentencePiece is a tokenizer and detokenizer framework that can train directly from raw sentences. Its published implementation supports both BPE and unigram-model segmentation. SentencePiece is therefore not another name for the unigram algorithm, and use of the library does not identify which of its supported algorithms a model used.^[19]

Consequences of Tokenization

Tokenization changes several properties at once:

• Sequence length determines how much visible text fits into a fixed token context.
• Vocabulary size affects embedding and output-layer storage.
• Segmentation determines which strings share parameters directly.
• Normalization and fallback rules determine whether every input can be represented.
• Token frequency affects the number of training updates received by each representation.
• Sequence-processing cost measured per token can differ across languages and scripts.

The survey by Mielke and colleagues traces word, character, hybrid, subword, byte, and other open-vocabulary approaches and finds no single best unit for all tasks. Tokenization choices trade shorter sequences against vocabulary size, linguistic fit, robustness, and implementation cost.^[20]

Tokenization also constrains evaluation. A model that divides the same sentence into more tokens averages log probability over a different number of events. Raw token-level loss or perplexity is therefore not directly comparable across substantially different tokenizers. Comparisons can use a shared tokenizer, report a common unit such as bytes or characters when valid, or evaluate the downstream task itself.^[5][29]

Training and Adaptation

Corpus Construction

Training begins with a corpus and a sampling policy. Common operations include text extraction, language identification, normalization, filtering, deduplication, domain balancing, train-validation-test splitting, and packing sequences into model contexts. Each operation changes the empirical distribution the model learns. A filtered corpus is not a neutral copy of "the internet"; it is a dataset produced by sources, collection dates, software, thresholds, and policy choices.^[31]

During self-supervised learning, prediction targets are derived from the data itself, such as the next token or a masked span. "Self-supervised" describes how targets are constructed, not whether humans influenced the source texts, filters, architecture, or evaluation. It also does not mean the corpus is free of duplicated benchmarks or machine-generated text.^[31]

Optimization usually processes mini-batches and updates parameters with gradient-based methods. Training reports need enough information to reconstruct the model, tokenizer, corpus mixture, number of tokens or updates, context length, optimizer, learning-rate schedule, precision, and randomization. A parameter count alone does not determine either training compute or performance.^[29]

Transfer and Fine-Tuning

Pretraining can create representations or conditional distributions that are reused in new settings. Transfer learning separates broad initial training from adaptation to a target domain or task. ULMFiT demonstrated a staged language-model fine-tuning procedure for text classification in 2018, while ELMo, GPT, and BERT supplied other influential forms of pretrained transfer.^{[11][12][13][21]}

Full fine tuning updates all or most pretrained parameters on target data and can require substantial memory. Parameter-efficient methods instead train a smaller set of added or selected parameters. LoRA freezes the base weights and learns low-rank update matrices in selected layers; its original experiments tested several transformer models and tasks. It is one adaptation method, not a guarantee that a model will preserve every base capability or match full fine-tuning in every setting.^[23]

Instruction tuning uses examples expressed as natural-language tasks and desired responses. Training with human preferences can add supervised demonstrations, a learned preference or reward model, and optimization against that signal. The InstructGPT study found that this process changed helpfulness and safety-related behavior on its tested prompt distribution, while also reporting remaining mistakes and limitations. Reinforcement Learning from Human Feedback (RLHF) is therefore a post-training family, not part of the definition of a language model.^[22]

Prompt engineering changes the conditioning context without changing model weights. In-context examples can elicit behaviors learned during training, but they do not create a durable parameter update and do not prove that the model inferred the intended rule. Prompt wording, ordering, separators, and answer format can all affect measured performance.^[29]

Retrieval-Augmented Generation adds documents or other records to the model's context at use time. Retrieval can provide current or source-specific evidence that is absent from the parameters. The final output still depends on retrieval quality, context construction, model behavior, and citation handling. Adding retrieval does not by itself make every generated claim supported.

Generation and Decoding

An autoregressive model supplies a probability distribution over the next token. A decoding procedure turns successive distributions into an output. The procedure is external to the learned probability function, and two decoders can produce different text from the same model and prompt.

Greedy decoding selects the highest-probability next token at each step. It is locally optimal but need not find the highest-probability complete sequence. Beam search keeps several partial sequences and is useful when the output is constrained by an input, as in some translation or recognition settings. For open-ended generation, aggressive probability maximization can produce generic or repetitive text.^[24]

Sampling draws a token from the model distribution. Temperature rescales logits before normalization: lower values concentrate probability on higher-scoring tokens, while higher values flatten the distribution. Top-k sampling restricts the choice to a fixed number of tokens. Nucleus, or top-p, sampling uses the smallest high-probability set whose cumulative probability reaches a threshold. Holtzman and colleagues introduced nucleus sampling after documenting degeneration from maximization-based decoding in their tested open-ended generation settings.^[24]

No decoder is universally best. Directed tasks may reward a narrow search toward a reference or constraint, while creative tasks may value diversity. Safety-sensitive systems may impose blocked tokens, grammars, validators, tool calls, or human review. Such controls alter the output process and should be reported separately from the base model.

Stopping is also part of decoding. Generation may end at an end-of-sequence token, a length limit, a delimiter, or an application rule. Length penalties and stopping conventions can change sequence rankings. Reproducible generation therefore requires the prompt, model revision, tokenizer, decoding method, parameters, seed where applicable, and termination rule.

Evaluation

Cross-Entropy and Perplexity

For a causal model evaluated on T target tokens, mean negative log likelihood is:

NLL = -(1 / T) sum over t of log P(x_t | x_1, ..., x_(t-1))

When natural logarithms are used, perplexity is exp(NLL). Equivalently, it is the geometric mean inverse probability assigned to the observed next tokens. Lower perplexity means the model assigned higher probability to that tokenized test sequence.^[5]

Perplexity is an intrinsic fit measure, not a percentage correct and not a complete ability score. It is most interpretable when models share the test data, tokenization, context construction, and likelihood calculation. Different vocabularies, byte fallback, normalization, context windows, or handling of document boundaries can change the value. A model can also have lower perplexity while performing worse on a task that depends on facts, instructions, robustness, or calibrated decisions.^[5][29]

Masked models require different scoring. Repeatedly masking each position produces pseudo-log-likelihood; exponentiating its average produces pseudo-perplexity. This can rank sentences or rescore candidate outputs, but it is not numerically interchangeable with left-to-right perplexity.^[16]

Evaluation data must be separate from the data used to fit parameters or select hyperparameters. A development set supports model choices; a held-out test set estimates performance after those choices are fixed. Repeated public comparison can indirectly turn a test set into a development target even without direct gradient training.

Task and Behavioral Evaluation

Extrinsic evaluation measures performance in an application. Depending on the task, metrics may include word error rate for recognition, accuracy or F1 for classification and question answering, retrieval measures, translation scores, factual support, calibration, latency, memory, or human judgments. Each metric operationalizes a narrower question than "does the model understand language?"

A benchmark combines tasks, datasets, prompts or protocols, and metrics. HELM argued for evaluating scenarios with multiple measures, including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. Its central lesson is methodological: model comparisons expose tradeoffs only when systems are evaluated under shared, transparent conditions and on more than one desired property.^[28]

Human evaluation is useful for qualities that automatic metrics capture poorly, but it is not automatically objective. Results depend on the evaluator population, instructions, interface, examples, rating scale, compensation, randomization, and agreement. A publication should state what people judged and how judgments were aggregated rather than report an unlabeled "human preference" score.^[28][29]

Reproducibility requires more than naming a model. Biderman and colleagues document sensitivity to prompt format, task implementation, model interface, likelihood normalization, few-shot selection, and other seemingly minor choices. A defensible comparison records exact model and tokenizer revisions, templates, decoding settings, software versions, data versions, and uncertainty or repeated runs where relevant.^[29]

Contamination and Memorization

Data contamination occurs when evaluation material or closely related information appears in training data. It can enter through benchmark pages, code repositories, papers, mirrors, discussion forums, or duplicated web documents. Overlap does not prove that a model used the answer during evaluation, but it weakens a test's claim to measure generalization to unseen examples.

Magar and Schwartz distinguished memorization of contaminated examples from exploitation that improves downstream task scores. Their controlled experiments found that one can occur without the other and that duplication and model size affect the results. This is why an overlap count alone neither proves a score invalid nor resolves the problem.^[30]

Detecting contamination is difficult when training data is undocumented or inaccessible. Mitigations include timestamped or private test sets, deduplication, search over disclosed corpora, renewed questions, and reporting uncertainty. None is perfect. Evaluation claims should state what was checked and avoid the absolute label "contamination-free" unless the construction actually supports it.

Scaling and Data

Empirical Scaling Laws

Scaling-law studies fit empirical relationships between model loss and resources such as parameter count, training data, and compute. Kaplan and colleagues reported approximate power-law trends for autoregressive transformer losses across the regimes they tested. These were measured relationships, not a theorem that all architectures, datasets, and downstream capabilities improve at the same rate.^[25]

Compute must be allocated between model size and the number of training tokens. Hoffmann and colleagues trained more than 400 models in their compute-optimal study and concluded that, within their setup, parameters and training tokens should grow in roughly equal proportions as compute increases. Their 70-billion-parameter Chinchilla model used the same reported training compute as the larger Gopher model but more training data and performed better across their evaluated tasks.^[26]

These findings corrected an allocation choice, not a permanent universal ratio. Corpus quality, architecture, optimizer, inference budget, target task, data reuse, and hardware efficiency all affect a practical optimum. Results fitted to one family and loss range should not be extrapolated without validation.

Available unique data can also constrain scaling. Muennighoff and colleagues studied repeated data and found that, for their fixed-compute experiments, up to four epochs of repetition produced negligible loss changes compared with unique data, while the value of further repeated tokens eventually decayed. They proposed scaling laws for this data-constrained regime and released the models and datasets from 400 runs.^[27]

Data Quality and Documentation

More tokens do not remove the need to understand their sources. Dodge and colleagues examined C4, a web corpus used in T5, and found material from unexpected domains, machine-generated text, benchmark examples, and filtering effects that disproportionately removed text about some minority identities. Their study illustrates how collection and cleaning choices shape both coverage and harms.^[31]

Data documentation should record source snapshots, licenses or terms where known, filtering, language identification, deduplication, removals, mixtures, and known gaps. Exact disclosure may be limited by privacy, contracts, or data volume, but absence of documentation is itself a limitation for reproducibility and risk analysis.

Scale can improve average predictive performance while leaving uneven behavior across domains and languages. A larger model trained on the same skewed distribution does not automatically repair missing coverage. Model size, data diversity, data quality, and evaluation design are separate variables.

Applications

Language models can be used wherever a system must predict, rank, represent, or generate language. Their role varies by application:

Scroll sideways for more →

The early speech and translation systems used language probability as one component in a larger decoder.^[3][4] Modern pretrained models often combine representation learning and conditional generation within one parameter set, but deployed applications still include retrieval systems, validators, policy layers, user interfaces, and external tools. Performance attributed to "the language model" may actually describe the whole system.

Limitations and Risks

Likelihood Is Not Truth

A model trained to imitate observed text can reproduce both accurate and inaccurate patterns. It may produce an hallucination, contradict supplied evidence, or express unwarranted certainty because token probability is not a truth predicate. TruthfulQA was designed around misconceptions that people may state in text; its experiments showed that strong likelihood-based models could imitate those false answers. The reported model rankings were specific to the tested systems and should not be generalized into a timeless rule about model size.^[35]

Retrieval, supervised examples, preference training, tool use, and verification can improve factual behavior, but each introduces another failure surface. A retrieval system can return irrelevant material, a model can misread a source, and a citation can fail to support the attached claim. High-stakes use requires domain-appropriate evidence and oversight rather than reliance on fluency.

Distribution Shift and Coverage

Language models usually perform best on distributions resembling their effective training data. Changes in subject matter, genre, time, dialect, script, or interaction pattern can degrade predictions. Aggregate test scores can hide poor performance on small groups or rare linguistic forms.

Coverage is shaped by corpus composition and tokenizer efficiency. Underrepresented languages may receive less data and longer token sequences. Domain adaptation can help, but it can also narrow behavior or introduce new errors. Claims of multilingual or domain-general capability should identify the languages, tasks, and evaluation populations actually tested.

Bias, Toxicity, and Social Context

Language data encodes social hierarchies, stereotypes, abuse, and disagreements about acceptable speech. Measuring AI bias requires a stated account of which behavior harms whom and in what setting. Blodgett and colleagues found that much NLP bias research used inconsistent motivations and metrics, recommending explicit normative reasoning and engagement with affected communities.^[32]

Gehman and colleagues showed that several pretrained models in their study could produce toxic continuations even from prompts scored non-toxic, and that none of their tested control methods was failsafe. The result supports testing both prompts and generated continuations under realistic conditions. It does not make one automated toxicity classifier a universal definition of harm.^[33]

Filtering can reduce some outputs while erasing discussion by or about marginalized groups. Safety evaluation should therefore consider false positives, false negatives, language variation, and who bears the cost of an error. A lower aggregate toxicity score can coexist with worse access or representation.

Memorization and Privacy

Generalization and memorization can coexist. Carlini and colleagues demonstrated a training-data extraction attack against GPT-2 and recovered hundreds of verbatim sequences, including material with identifying information. Their experiment did not show that every prompt reveals arbitrary training data, but it established that generative models can expose memorized examples under a suitable attack.^[34]

Risk depends on duplication, model and training choices, query access, and the sensitivity of the corpus. Data minimization, deduplication, privacy review, access controls, output monitoring, and privacy-preserving training can reduce exposure. Public availability of a source does not eliminate privacy or security concerns.

Reproducibility, Control, and Resource Cost

Closed training corpora, unavailable weights, changing hosted endpoints, and undisclosed post-training make independent reproduction difficult. Even open weights do not fully reproduce a model without the data, tokenizer, code, and training configuration. Evaluation should separate results reproduced from public artifacts from values copied from a provider report.^[29]

Generated behavior is also sensitive to prompt and decoder choices. A refusal, unsafe answer, or correct answer observed once does not estimate its probability across realistic inputs. Robustness testing uses varied prompts, seeds, attacks, and contexts and reports the distribution of outcomes.

Language-model development can impose computational, environmental, and labor costs. The risk taxonomy by Weidinger and colleagues groups observed and anticipated harms across discrimination, misinformation, malicious use, human-computer interaction, automation, access, and environmental effects. It presents a framework for investigation rather than a claim that every language model produces every listed harm.^[36]

AI safety measures are consequently layered: corpus governance, model testing, secure deployment, access controls, monitoring, incident response, and application-specific human oversight. No single benchmark, filter, or post-training method certifies a model as safe for all uses.

Relationship to Adjacent Concepts

Scroll sideways for more →

The boundaries are functional rather than mutually exclusive. BERT is a transformer-based masked language model and a foundation model used for transfer, but it is not ordinarily a left-to-right generator. A decoder-only model can be a causal language model, a large language model, a foundation model, and part of a generative-AI system at the same time. The labels answer different questions about objective, architecture, scale, reuse, and application.^[13][37]

References

1. ^Claude E. Shannon. "A Mathematical Theory of Communication." Bell System Technical Journal, 1948. doi.org/...j.1538-7305.1948.tb01338.x
2. ^Claude E. Shannon. "Prediction and Entropy of Printed English." Bell System Technical Journal, 1951. doi.org/...j.1538-7305.1951.tb01366.x
3. ^Lalit R. Bahl, Frederick Jelinek, and Robert L. Mercer. "A Maximum Likelihood Approach to Continuous Speech Recognition." IEEE Transactions on Pattern Analysis and Machine Intelligence, 1983. research.ibm.com/...-continuous-speech-recognition
4. ^Peter F. Brown et al. "A Statistical Approach to Machine Translation." Computational Linguistics, 1990. aclanthology.org/J90-2002
5. ^Daniel Jurafsky and James H. Martin. Speech and Language Processing, third-edition online manuscript, 2026 release. web.stanford.edu/...ed3book.pdf
6. ^Stanley F. Chen and Joshua Goodman. "An Empirical Study of Smoothing Techniques for Language Modeling." Harvard Computer Science Technical Report TR-10-98, 1998. dash.harvard.edu/...25104739
7. ^Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. "A Neural Probabilistic Language Model." Journal of Machine Learning Research, 2003. jmlr.org/...bengio03a
8. ^Sepp Hochreiter and Jürgen Schmidhuber. "Long Short-Term Memory." Neural Computation, 1997. direct.mit.edu/...Long-Short-Term-Memory
9. ^Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. "Recurrent Neural Network Based Language Model." Interspeech, 2010. isca-archive.org/...mikolov10_interspeech
10. ^Ashish Vaswani et al. "Attention Is All You Need." Advances in Neural Information Processing Systems 30, 2017. proceedings.neurips.cc/...bd053c1c4a845aa-Abstract
11. ^Matthew E. Peters et al. "Deep Contextualized Word Representations." NAACL-HLT, 2018. aclanthology.org/N18-1202
12. ^Alec Radford et al. "Improving Language Understanding by Generative Pre-Training." OpenAI, 2018. cdn.openai.com/...language_understanding_paper.pdf
13. ^Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL-HLT, 2019. aclanthology.org/N19-1423
14. ^Mike Lewis et al. "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension." ACL, 2020. aclanthology.org/2020.acl-main.703
15. ^Colin Raffel et al. "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." Journal of Machine Learning Research, 2020. jmlr.org/...20-074
16. ^Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. "Masked Language Model Scoring." ACL, 2020. aclanthology.org/2020.acl-main.240
17. ^Rico Sennrich, Barry Haddow, and Alexandra Birch. "Neural Machine Translation of Rare Words with Subword Units." ACL, 2016. aclanthology.org/P16-1162
18. ^Taku Kudo. "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates." ACL, 2018. aclanthology.org/P18-1007
19. ^Taku Kudo and John Richardson. "SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing." EMNLP System Demonstrations, 2018. aclanthology.org/D18-2012
20. ^Sabrina J. Mielke et al. "Between Words and Characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP." arXiv, 2021. arxiv.org/...2112.10508
21. ^Jeremy Howard and Sebastian Ruder. "Universal Language Model Fine-tuning for Text Classification." ACL, 2018. aclanthology.org/P18-1031
22. ^Long Ouyang et al. "Training Language Models to Follow Instructions with Human Feedback." Advances in Neural Information Processing Systems 35, 2022. proceedings.neurips.cc/...14f58805a001731-Abstract
23. ^Edward J. Hu et al. "LoRA: Low-Rank Adaptation of Large Language Models." International Conference on Learning Representations, 2022. openreview.net/forum
24. ^Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. "The Curious Case of Neural Text Degeneration." International Conference on Learning Representations, 2020. openreview.net/forum
25. ^Jared Kaplan et al. "Scaling Laws for Neural Language Models." arXiv, 2020. arxiv.org/...2001.08361
26. ^Jordan Hoffmann et al. "An Empirical Analysis of Compute-Optimal Large Language Model Training." Advances in Neural Information Processing Systems 35, 2022. proceedings.neurips.cc/...a3e5-Abstract-Conference
27. ^Niklas Muennighoff et al. "Scaling Data-Constrained Language Models." Advances in Neural Information Processing Systems 36, 2023. proceedings.neurips.cc/...c196-Abstract-Conference
28. ^Percy Liang et al. "Holistic Evaluation of Language Models." Transactions on Machine Learning Research, 2023. arxiv.org/...2211.09110
29. ^Stella Biderman et al. "Lessons from the Trenches on Reproducible Evaluation of Language Models." arXiv, 2024. arxiv.org/...2405.14782
30. ^Inbal Magar and Roy Schwartz. "Data Contamination: From Memorization to Exploitation." ACL, 2022. aclanthology.org/2022.acl-short.18
31. ^Jesse Dodge et al. "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus." EMNLP, 2021. aclanthology.org/2021.emnlp-main.98
32. ^Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. "Language (Technology) Is Power: A Critical Survey of 'Bias' in NLP." ACL, 2020. aclanthology.org/2020.acl-main.485
33. ^Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. "RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models." Findings of EMNLP, 2020. aclanthology.org/2020.findings-emnlp.301
34. ^Nicholas Carlini et al. "Extracting Training Data from Large Language Models." USENIX Security Symposium, 2021. usenix.org/...carlini-extracting
35. ^Stephanie Lin, Jacob Hilton, and Owain Evans. "TruthfulQA: Measuring How Models Mimic Human Falsehoods." ACL, 2022. aclanthology.org/2022.acl-long.229
36. ^Laura Weidinger et al. "Taxonomy of Risks Posed by Language Models." ACM Conference on Fairness, Accountability, and Transparency, 2022. doi.org/...3531146.3533088
37. ^Rishi Bommasani et al. "On the Opportunities and Risks of Foundation Models." Stanford Center for Research on Foundation Models, 2021. arxiv.org/...2108.07258
38. ^Albert Gu and Tri Dao. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." Conference on Language Modeling, 2024. openreview.net/forum
39. ^Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. "Structured Denoising Diffusion Models in Discrete State-Spaces." Advances in Neural Information Processing Systems 34, 2021. proceedings.neurips.cc/...e97125b70e6973d-Abstract
40. ^Subham Sekhar Sahoo et al. "Simple and Effective Masked Diffusion Language Models." Advances in Neural Information Processing Systems 37, 2024. proceedings.neurips.cc/...e0ad-Abstract-Conference