Bradley-Terry model
The Bradley-Terry model is a probability model for paired comparisons: it gives each item in a set a positive strength and says that the chance of item $i$ beating (or being preferred to) item $j$ equals $i$'s strength divided by the sum of both strengths. The model was first introduced by E. Zermelo in a 1929 paper on computing tournament results and is named after Ralph Allan Bradley and Milton E. Terry, who rediscovered it in a 1952 Biometrika paper on paired-comparison experiments [1][2][3]. Written on a log scale, it is a logistic regression on the difference between two scores, which makes it a close relative of the Elo rating system [3][9]. In artificial intelligence it has two prominent uses: it is the usual way to turn human preference labels into a reward model for reinforcement learning from human feedback (RLHF), and it is the statistical model behind the Chatbot Arena leaderboard, which ranks large language models from crowdsourced head-to-head votes [9][13][14].
Definition
Suppose $N$ items (players, teams, products, model outputs) are compared in pairs, and each comparison has a winner. The Bradley-Terry model assigns item $i$ a strength $\pi_i > 0$ and models the probability that $i$ beats $j$ as [3][5]:
Writing $\pi_i = e^{s_i}$, with $s_i$ a real-valued score, gives the equivalent form [3]:
where $\sigma$ is the logistic sigmoid function. The win probability depends only on the difference between the two scores, so the model is a logistic model of score differences [3]. Zermelo, writing in German, called the parameters "Spielstärken" (playing strengths); other authors call them worth, skill or merit parameters, ratings or weights [3].
Multiplying every $\pi_i$ by the same constant (equivalently, adding a constant to every $s_i$) leaves all probabilities unchanged, so one parameter has to be pinned down by a normalization [3]. One convenient choice fixes the geometric mean of the strengths at 1. Under that choice, an item's strength $\pi$ equals its odds of beating an average item, because the probability of beating an item of strength 1 is $\pi / (\pi + 1)$ [3]. The Chatbot Arena paper instead fixes the first model's coefficient at zero [14].
History
| Year | Work | Contribution |
|---|---|---|
| 1929 | E. Zermelo, Mathematische Zeitschrift | Introduces the model to compute tournament results and gives an iterative maximum likelihood algorithm [2][3] |
| 1952 | Ralph A. Bradley and Milton E. Terry, Biometrika | Rediscover the model in "Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons" [1][3] |
| 1957 | L. R. Ford, American Mathematical Monthly | Conditions for a unique maximum likelihood solution and a convergence proof [3][4] |
| 1967 | P. V. Rao and L. L. Kupper, JASA | Extension that models ties [3][7] |
| 1970 | Roger R. Davidson, JASA | Alternative extension for ties [3][8] |
| 2004 | David R. Hunter, Annals of Statistics | MM algorithms for generalized Bradley-Terry models [5][6] |
| 2017 | Christiano et al. | Bradley-Terry loss used to learn a reward function from human preferences in deep RL [9] |
| 2024 | LMSYS | Chatbot Arena leaderboard moves from online Elo to a Bradley-Terry fit (announced December 2023) [15][18] |
Zermelo's paper, "Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung," appeared in volume 29 of Mathematische Zeitschrift in 1929 [2]. It both introduced the model and described the iterative procedure that is still the standard way of fitting it [3]. Bradley and Terry, then at the Virginia Agricultural Experiment Station of the Virginia Polytechnic Institute, published their version in Biometrika in December 1952, and the model takes its common name from that paper [1][3]. The literature grew quickly: a bibliography on paired comparisons published in 1976 by Davidson and Farquhar already listed several hundred entries [5].
Applications named in the statistical literature include rankings in sports, chess and other games, paired-comparison studies of consumer choice, and studies of dominance behavior in animals and humans [3]. It is also a standard way to build a multiclass classifier out of the outputs of many binary classifiers, an approach associated with Hastie and Tibshirani (1998) [5].
Estimation
Maximum likelihood
Let $w_{ij}$ be the number of times $i$ beat $j$. The log-likelihood of the observed results is [3]:
Setting the derivative to zero does not give a closed-form answer, but Zermelo showed that the maximum likelihood estimate can be found by repeating the update [3]:
The numerator is simply item $i$'s total number of wins. Starting from any positive values and iterating converges, under conditions discussed below, to the global maximum of the likelihood [3]. The resulting strengths can be sorted into a ranking or used directly as ratings [3].
Newman notes that the likelihood is concave, so standard convex optimization methods can also be used [3]. Newton's method is often fastest for a handful of items, but its cost per iteration grows as $N^3$, which becomes prohibitive for large problems [3]. Hunter (2004) showed that Zermelo's iteration is a special case of a minorization-maximization (MM) algorithm and derived MM algorithms for generalized versions of the model [3][5][6]; Caron and Doucet later reinterpreted those MM algorithms as expectation-maximization algorithms and used the same latent variables to build Gibbs samplers for Bayesian inference [5]. Spectral methods based on random walks over the comparison network approximate the maximum likelihood solution without maximizing the likelihood directly [3].
Zermelo's iteration is known to converge slowly [3]. In a paper published in the Journal of Machine Learning Research in 2023, M. E. J. Newman proposed regrouping the same equation as
and proved that it converges to the same solution; in his tests it was over a hundred times faster than Zermelo's algorithm in some cases [3].
Existence of the estimate
The maximum likelihood estimate does not always exist. Zermelo and Ford showed that the likelihood has a single stationary point, the global maximum, when the directed network of results is strongly connected: for every pair of items there must be a chain of wins leading from one to the other [3][4]. If the network is not strongly connected, there is no maximum and the scores diverge. An item that wins every one of its comparisons, for example, has an estimated strength of infinity, and one that loses every comparison gets zero [3]. One remedy is to place a prior on the scores and compute a maximum a posteriori estimate instead; Newman argues for a logistic prior, which corresponds to assuming that the probability of beating an average item is uniformly distributed between 0 and 1 [3].
As logistic regression
Because $P(i \succ j) = \sigma(s_i - s_j)$, fitting the model is a binary classification problem: each comparison is an example, the features encode which two items took part, and the loss is the binary cross-entropy [14]. The Chatbot Arena paper defines its Bradley-Terry coefficients in exactly this way, as the vector that minimizes expected binary cross-entropy [14], and the open-source FastChat code that LMSYS published fits that loss numerically with the L-BFGS-B optimizer [16]. Framing the model as a regression also makes it easy to add covariates, which Chatbot Arena later did for response style [17].
Extensions
| Extension | What it adds | Source |
|---|---|---|
| Ties counted as half wins | A draw is half a win for each side; used in the Elo chess system and by Chatbot Arena | [3][15] |
| Rao-Kupper model (1967) | Ties become a separately modeled outcome | [3][7] |
| Davidson model (1970) | A tie parameter $\nu$; ties are likeliest between evenly matched items | [3][8] |
| Home advantage | Adjusts for home advantage | [5] |
| Plackett-Luce model | Rankings or choices among more than two items | [5][13] |
| Team comparisons | Comparisons between teams | [5] |
| Covariates (style control) | Extra regression terms for features of each comparison | [17] |
The simplest treatment of a draw counts it as half a win and half a loss for each side, which only requires changing the win counts $w_{ij}$ [3]. Models that treat a tie as its own outcome go further. In Davidson's version, written here in the parametrization Newman uses, with $\nu > 0$ controlling how often ties happen [3]:
The tie probability is largest when the two items are evenly matched and falls toward zero as their strengths move apart [3].
The Plackett-Luce model generalizes Bradley-Terry from pairs to full rankings or choices among several items, and is used for ranking multiple individuals and in choice models [5]. Christiano and colleagues described their Bradley-Terry preference model as the specialization of the Luce-Shepard choice rule to preferences over trajectory segments [9], and the Direct Preference Optimization paper notes that its derivation also works under Plackett-Luce models when several ranked answers are available [13].
Relation to the Elo rating system
The Elo rating system, developed for chess [9], predicts the expected score of player A against player B as follows, in the form implemented in LMSYS's FastChat code [16]:
After each game the ratings move by $K(S_A - E_A)$, where $S_A$ is the actual result (1, 0.5 or 0) and $K$ is a step size [16]. The prediction has the same logistic form as the Bradley-Terry model, with base 10 and a scale of 400 in place of base $e$: a Bradley-Terry score $s$ maps to an Elo-style rating $R = 400,s/\ln 10$ plus an arbitrary offset [16]. Christiano and colleagues drew the same parallel in 2017, writing that their Bradley-Terry reward model was analogous to Elo, where the difference in points between two players estimates the probability that one defeats the other [9].
The two differ in how ratings are estimated. Elo updates ratings one game at a time, so recent results weigh more and the final ratings depend on the order in which games were played; it is designed to track a player whose strength changes [15]. A Bradley-Terry fit assumes fixed strengths, treats game order as irrelevant, and estimates all strengths at once by maximizing the likelihood of every observed result [15]. That difference is the reason Chatbot Arena gave for changing methods (see below).
Use in reward modeling and RLHF
Learning rewards from preferences
In "Deep reinforcement learning from human preferences" (2017), Paul Christiano and co-authors trained agents on Atari games and simulated robot locomotion tasks without access to the true reward, using human comparisons of short trajectory clips on less than 1% of the agent's interactions [9]. They modeled the probability that a human prefers one clip over another as a Bradley-Terry function of the reward predicted for each clip (summed over the clip), and fit the reward predictor by minimizing the cross-entropy between those predictions and the human labels [9]. The paper describes this as following the Bradley-Terry model for estimating score functions from pairwise preferences [9].
The same recipe carried over to language models. In "Learning to summarize from human feedback" (2020), the reward model starts from a supervised baseline with a randomly initialized linear head that outputs a scalar, and is trained on comparisons between two summaries with the loss [10]:
where $x$ is the input, $y_w$ the preferred output and $y_l$ the other one. This is the negative log-likelihood of the Bradley-Terry model with the reward $r_\theta(x, y)$ playing the role of the score [10][13].
| Work | Year | How it uses the Bradley-Terry loss |
|---|---|---|
| Christiano et al. | 2017 | Preferences between trajectory clips in Atari and robot locomotion [9] |
| Stiennon et al. | 2020 | Pairwise comparisons of summaries; rewards normalized so reference summaries average 0 [10] |
| InstructGPT (Ouyang et al.) | 2022 | Labelers rank 4 to 9 responses; all $\binom{K}{2}$ pairs from one prompt form a single batch element [11] |
| Llama 2 (Touvron et al.) | 2023 | Same binary ranking loss plus a margin $m(r)$ that grows with how strongly annotators preferred one response [12] |
| DPO (Rafailov et al.) | 2023 | Bradley-Terry likelihood rewritten in terms of the policy itself, with no separate reward model [13] |
InstructGPT's authors reported that shuffling all pairwise comparisons into one dataset made the reward model overfit after a single pass, because comparisons drawn from the same ranking are highly correlated; training on all of a prompt's comparisons as one batch element fixed this and was cheaper, since each completion needs only one forward pass [11]. Llama 2's reward models were trained with "a binary ranking loss consistent with Ouyang et al. (2022)" and a margin term derived from a four-point preference rating; the authors reported that the margin improved helpfulness reward model accuracy, especially on pairs where the two responses were more clearly different [12].
Direct Preference Optimization
Direct Preference Optimization (DPO), introduced by Rafailov and colleagues in 2023, keeps the Bradley-Terry assumption but removes the explicit reward model [13]. Its derivation relies on the fact that Bradley-Terry probabilities depend only on the difference between two rewards. Expressing the reward of the optimal KL-regularized policy in terms of the policy and a reference model, the normalizing constant cancels in that difference, and the Bradley-Terry likelihood becomes a loss on the policy itself [13]:
Here $\pi_\theta$ is the model being trained, $\pi_{\text{ref}}$ the reference model and $\beta$ the strength of the KL divergence penalty. The authors describe the procedure as equivalent to fitting a reparametrized Bradley-Terry model [13].
Use in LLM leaderboards
Chatbot Arena, whose leaderboard is now published by the company Arena (called LMArena until January 2026), shows a user two anonymous model responses to the same prompt and asks which is better [14][23]. Its first leaderboards reported online Elo ratings. In a December 2023 post, LMSYS researchers wrote that they had noticed "considerable variability in the ratings using the classic online algorithm" and that their earlier bootstrap confidence intervals were too wide [15]. They moved to the maximum likelihood estimate of a Bradley-Terry model, which assumes each model's performance does not change and so makes the ratings independent of vote order; ties are counted as half a win and half a loss, and confidence intervals come from bootstrapping the fit [15]. The public leaderboard dataset's changelog dates the rating-system change to 2024-01-09 [18]. The 2024 Chatbot Arena paper explains the switch by saying "the BT coefficients are better for the purpose of statistical estimation" [14].
To keep familiar numbers, the published scores are put on an Elo-like scale. In LMSYS's FastChat code, the fitted coefficients are multiplied by 400 and offset from 1,000, then shifted so that a reference model (Mixtral-8x7B-Instruct v0.1) sits at 1,114 [16]. A 400-point gap therefore corresponds to 10-to-1 odds, as in chess Elo [16].
Several statistical refinements followed:
- Uncertainty. The paper compares bootstrap confidence intervals with "sandwich" robust standard errors and chose the sandwich intervals because they are smaller in large samples [14]. It notes that maximum likelihood estimates remain asymptotically normal even when the Bradley-Terry assumptions do not hold, as long as the sandwich covariance is used [14].
- Sampling. Because some model pairs are shown more often than others, the estimator weights each vote by the inverse of the probability that its pair was sampled, so that the score targets a uniform distribution over pairs [14]. The leaderboard changelog records frequency-based re-weighting from 2025-07-23 [18].
- Style control. In August 2024, LMSYS added "extra, additive style coefficients" to the Bradley-Terry regression for differences in response length and Markdown formatting, with length entering as the difference between the two responses' token counts divided by their sum, so the model coefficients are estimated with the effect of style held fixed [17]. Length turned out to be the dominant style factor [17]. Style control became the default for the text and vision arenas on 2025-05-16 [18].
- Non-transitivity. An appendix to the paper describes a nonparametric extension of the model for cases where the ranking is not necessarily transitive [14].
In December 2025 LMArena released Arena-Rank, an Apache 2.0 licensed Python package that implements the Bradley-Terry model, the contextual extension used for style control and the re-weighting step, and that computes confidence intervals in closed form rather than by bootstrapping [19]. The Elo rating system article covers the Arena's rating history and criticisms of the leaderboard, including The Leaderboard Illusion, in more detail.
Limitations and criticism
One score per item. A Bradley-Terry model gives each item a single number, so it cannot represent preferences that cycle (A preferred to B, B to C, C to A). Researchers at Google DeepMind cited this as a reason to learn a pairwise preference model instead of a reward model in "Nash Learning from Human Feedback" (2023) [21]. They noted that whether individual humans have non-transitive preferences is a long-studied question, but that even if each individual is transitive, preferences averaged over a population need not be [21]. They also argued that maximizing an Elo or Bradley-Terry score is not always the same as maximizing the probability of beating a population of opponents, even when preferences fit the model exactly [21].
Deterministic preferences. In a 2023 analysis, Azar and colleagues pointed out that if one output is always preferred to another, the Bradley-Terry model requires the gap between their rewards to be infinite [20]. When this is fed into the KL-regularized objective used by RLHF and DPO, the regularization stops having any effect; with small datasets, an empirical preference of 1 can arise even when the true preference is 0.8, so DPO in particular can overfit [20]. They proposed an objective expressed directly in pairwise preferences that avoids the Bradley-Terry substitution [20].
Not the only option. Sun, Shen and Ton (2024) studied why a model built for matching competitors works for reward modeling when each prompt-response pair is compared only sparsely, and gave convergence-rate results for neural Bradley-Terry reward models [22]. They argued, however, that the model is not necessary for downstream optimization, because a reward model only needs to preserve the correct ordering up to a monotonic transformation, and they proposed a classification-based alternative, comparing approaches in more than 12,000 experimental setups across 6 base LLMs and 2 datasets [22].
Model assumptions in leaderboards. The Chatbot Arena authors note that the Bradley-Terry model assumes a parametric form for win rates, and rely on sandwich standard errors and a nonparametric extension to keep their estimates valid when that form is wrong [14]. Style control was added to separate the effect of presentation, chiefly answer length, from the model coefficients [17].
Continue reading
- Reward ModelA reward model (RM) is a model trained to score the outputs of another AI system, producing a scalar estimate of how good a candidate response is according to some standard, most often human preference.Mentioned 11 times above
- Chatbot ArenaChatbot Arena (now branded simply as Arena, and previously known as LMArena) is a crowdsourced evaluation platform for large language models that ranks AI systems based on human preferences through anonymous…Mentioned 10 times above
- Elo rating system (AI model ranking)The Elo rating system, as applied to AI models, is a method for turning a pile of head-to-head preference votes into a single number per model, so that large language models and chatbots can be ranked on a…Mentioned 4 times above
- Direct Preference Optimization (DPO)Direct Preference Optimization (DPO) is a method for aligning large language models with human preferences that replaces the multi-stage reinforcement learning from human feedback (RLHF) pipeline with a single…Mentioned 3 times above
- LMArenaLMArena is a crowdsourced artificial intelligence evaluation platform and company that ranks large language models by having anonymous users vote on which of two blind, side-by-side model responses they preferMentioned twice above
- Logistic RegressionLogistic regression is a statistical model for a binary response. It represents the conditional probability of one outcome as the logistic transformation of a linear predictor.Mentioned twice above
More from See also
More to read
Topics:Machine LearningModel EvaluationReinforcement LearningStatistics
References
- ^1 ^2 ^3Bradley, Ralph Allan; Terry, Milton E. "Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons". Biometrika 39 (3-4): 324-345, December 1952. doi.org/...39.3-4.324
- ^1 ^2 ^3Zermelo, E. "Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung". Mathematische Zeitschrift 29: 436-460, 1929. doi.org/...BF01180541
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35Newman, M. E. J. "Efficient computation of rankings from pairwise comparisons". Journal of Machine Learning Research 24, 238, 2023 (arXiv:2207.00076). arxiv.org/...2207.00076
- ^1 ^2Ford, L. R. "Solution of a Ranking Problem from Binary Comparisons". The American Mathematical Monthly 64 (8): 28-33, October 1957. doi.org/...2308513
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Caron, Francois; Doucet, Arnaud. "Efficient Bayesian Inference for Generalized Bradley-Terry Models". INRIA Research Report 7445 (arXiv:1011.1761), November 2010. arxiv.org/...1011.1761
- ^1 ^2Hunter, David R. "MM algorithms for generalized Bradley-Terry models". The Annals of Statistics 32 (1): 384-406, February 2004. doi.org/...1079120141
- ^1 ^2Rao, P. V.; Kupper, L. L. "Ties in Paired-Comparison Experiments: A Generalization of the Bradley-Terry Model". Journal of the American Statistical Association 62 (317): 194-204, March 1967. doi.org/...01621459.1967.10482901
- ^1 ^2Davidson, Roger R. "On Extending the Bradley-Terry Model to Accommodate Ties in Paired Comparison Experiments". Journal of the American Statistical Association 65 (329): 317-328, March 1970. doi.org/...01621459.1970.10481082
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Christiano, Paul; Leike, Jan; Brown, Tom B.; Martic, Miljan; Legg, Shane; Amodei, Dario. "Deep reinforcement learning from human preferences". arXiv:1706.03741, June 2017. arxiv.org/...1706.03741
- ^1 ^2 ^3Stiennon, Nisan; Ouyang, Long; Wu, Jeff; et al. "Learning to summarize from human feedback". arXiv:2009.01325, September 2020. arxiv.org/...2009.01325
- ^1 ^2Ouyang, Long; Wu, Jeff; Jiang, Xu; et al. "Training language models to follow instructions with human feedback". arXiv:2203.02155, March 2022. arxiv.org/...2203.02155
- ^1 ^2Touvron, Hugo; Martin, Louis; Stone, Kevin; et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models". arXiv:2307.09288, July 2023. arxiv.org/...2307.09288
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Rafailov, Rafael; Sharma, Archit; Mitchell, Eric; Ermon, Stefano; Manning, Christopher D.; Finn, Chelsea. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model". arXiv:2305.18290, May 2023. arxiv.org/...2305.18290
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Chiang, Wei-Lin; Zheng, Lianmin; Sheng, Ying; et al. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference". arXiv:2403.04132, March 2024. arxiv.org/...2403.04132
- ^1 ^2 ^3 ^4 ^5 ^6Chiang, Wei-Lin; Li, Tim; Gonzalez, Joseph E.; Stoica, Ion. "Chatbot Arena: New models & Elo system update". LMSYS Org blog, 2023-12-07. lmsys.org/...2023-12-07-leaderboard
- ^1 ^2 ^3 ^4 ^5 ^6LMSYS. "rating_systems.py" (FastChat source code, Elo and Bradley-Terry rating functions). GitHub, lm-sys/FastChat, accessed 2026-11-18. github.com/...rating_systems.py
- ^1 ^2 ^3 ^4 ^5Li, Tianle; Angelopoulos, Anastasios; Chiang, Wei-Lin. "Does style matter? Disentangling style and substance in Chatbot Arena". LMSYS Org blog, August 2024. lmsys.org/...2024-08-28-style-control
- ^1 ^2 ^3 ^4LMArena. "lmarena-ai/leaderboard-dataset" (dataset card, notes on rating-system changes). Hugging Face, accessed 2026-11-18. huggingface.co/...leaderboard-dataset
- ^Arena Team. "Arena-Rank: Open Sourcing the Leaderboard Methodology". Arena blog, 2025-12-18 (updated 2026-02-21). arena.ai/...arena-rank
- ^1 ^2 ^3Azar, Mohammad Gheshlaghi; Rowland, Mark; Piot, Bilal; Guo, Daniel; Calandriello, Daniele; Valko, Michal; Munos, Remi. "A General Theoretical Paradigm to Understand Learning from Human Preferences". arXiv:2310.12036, October 2023. arxiv.org/...2310.12036
- ^1 ^2 ^3Munos, Remi; Valko, Michal; Calandriello, Daniele; et al. "Nash Learning from Human Feedback". arXiv:2312.00886, December 2023. arxiv.org/...2312.00886
- ^1 ^2Sun, Hao; Shen, Yunyi; Ton, Jean-Francois. "Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives". arXiv:2411.04991, November 2024. arxiv.org/...2411.04991
- ^Arena Team. "LMArena is now Arena". Arena blog, 2026-01-28. arena.ai/...lmarena-is-now-arena
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
1 revision · v1 · 3,757 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check before publication on November 18, 2026: 74 claims checked against 21 cited and independent sources; 1 material and 2 minor corrections applied before publication.
Cite this page: AI Wiki. "Bradley-Terry model." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/bradley_terry_model
