# Leo Breiman

> Source: https://aiwiki.ai/wiki/leo_breiman
> Updated: 2026-07-24
> Fact-checked: 2026-07-24
> Categories: AI History, Machine Learning, People, Statistics
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Leo Breiman." aiwiki.ai, 24 Jul 2026. https://aiwiki.ai/wiki/leo_breiman
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Leo Breiman (January 27, 1928 to July 5, 2005) was an American statistician at [UC Berkeley](https://aiwiki.ai/wiki/uc_berkeley) whose work on tree-based prediction supplied [machine learning](https://aiwiki.ai/wiki/machine_learning) with several of its most durable tools. He was the first of the four authors of *Classification and Regression Trees* (1984), the monograph behind [CART](https://aiwiki.ai/wiki/cart); he invented [bagging](https://aiwiki.ai/wiki/bagging) in a 1994 Berkeley technical report; and his 2001 paper "Random Forests" introduced the [random forest](https://aiwiki.ai/wiki/random_forest), which remains a standard method for prediction on tabular data [1][2][3][4].

In the same year as the random forest paper he published "Statistical Modeling: The Two Cultures" in *Statistical Science*, an essay arguing that academic statistics had hobbled itself by assuming that data are generated by a stochastic model, while a rival "algorithmic modeling" culture growing up mostly outside the field was solving the problems that actually mattered [5]. Statisticians have been arguing with it ever since, including through a twentieth-anniversary set of commentaries in 2021 [18].

Breiman came to all of this late and sideways. He trained as a probabilist, quit a tenured mathematics job because he decided he was not cut out for abstract mathematics, spent thirteen years as a freelance consultant working on freeway traffic, ozone forecasts and court-system delay, and only returned to academia at Berkeley in 1980, in his early fifties [5][6][7].

## Early life and education

Breiman was born in New York City on January 27, 1928, the only child of Max and Lena Breiman, immigrants from Eastern Europe; his father worked as a tailor and sewing machine operator [6][7]. When he was five the family moved to California, first to San Francisco and then to Los Angeles, where he grew up in Boyle Heights and graduated from Roosevelt High School in 1945 [6][7].

He entered the California Institute of Technology and earned a degree in physics in 1949, though he later said his grades slid over his last three years because a diet of nothing but science and engineering had begun to put him off: "The place was like a scientific monastery" [6]. He went to Columbia University intending to switch to philosophy, was talked out of it by the head of the philosophy department, and took a master's degree in mathematics there in 1950 [6][7]. He supported himself between terms as a waiter in the Catskills and as a dishwasher in the Merchant Marine [6].

He completed his Ph.D. at UC Berkeley in 1954 with a dissertation titled "Homogeneous Processes" under Michel Loève [6][8]. His early research was in probability and information theory: the Shannon-Breiman-McMillan theorem dates from 1957, and a 1960 paper on optimal play in favorable games grew out of his interest in the Kelly criterion [6].

## UCLA, Liberia and thirteen years of consulting

After almost two years in the Army and a spell at Berkeley on an acting appointment, Breiman walked into the UCLA mathematics department off the street and asked for a job. He got one, and tenure fairly quickly, as the department's only probabilist teaching the graduate probability course [6]. He also volunteered to teach mathematics to two fifth-grade classes, one of them at an experimental school for emotionally disturbed children, on the theory that children were being taught to see mathematics as disconnected from the world around them [6][7].

He took a UNESCO sabbatical as an educational statistician in Liberia, where the job was to find out how many children were actually in the country's schools. With only about 50 miles of paved road nationwide and many schools deep in rain forest, he organized 20 teams to walk or paddle out and count the children in person [6][7].

After about seven years at UCLA he concluded, in his own words, that he was "not cut out to be an abstract mathematician," and resigned his tenured position [6]. He withdrew his retirement savings and spent six months doing nothing but working on the graduate textbook that appeared in 1968 as *Probability*, which the Society for Industrial and Applied Mathematics later approached him about reissuing in its classics series [1][6][7]. He wrote two further textbooks, *Probability and Stochastic Processes with a View Toward Applications* and *Statistics with a View Toward Applications* [1].

The consulting years that followed lasted thirteen [5][6]. He worked first for Systems Development Corporation on freeway traffic, then for Technology Services Corporation under Bill Meisel, who steered him toward classification and regression problems [6]. Clients and projects included the Environmental Protection Agency, the California State Air Pollution Board, the U.S. Department of Transportation, the California Transportation Department, and state and federal court systems [5][6]. The problem list he later published reads like a catalogue of what would become mainstream machine learning applications: next-day ozone prediction, identifying halogen-containing compounds from mass spectra, classifying ships from high-altitude radar returns and submarines from sonar returns, recognizing hand-sent Morse code, speech recognition, and the sources of delay in criminal trials [5]. In parallel he served on the Santa Monica School Board, eventually as its president [6][7].

Two of those consulting projects mattered most for what came later. The ozone project assembled seven years of hourly and daily readings on more than 450 meteorological variables in the Los Angeles Basin, an unusually high-dimensional problem for the 1970s; Breiman called it a failure, since the false alarm rate of the final predictor was too high [5]. The chlorine project, funded by the EPA to detect chlorine in compounds from mass spectra, produced a [decision tree](https://aiwiki.ai/wiki/decision_tree) built from a set of 1,500 yes-or-no questions that reached 95 percent accuracy on both chlorines and non-chlorines [5]. That work fed directly into CART.

## Berkeley and the Statistical Computing Facility

Peter Bickel invited Breiman back to Berkeley for a semester; a year later David Brillinger offered him a faculty position, and he joined the Department of Statistics in 1980 [6][7]. He immediately concluded that the department could not do serious applied work with the hardware it had: a PDP-11 with 32 kilobytes of main memory, with everything else run downstairs on the Evans Hall mainframe [6][7]. He talked the Office of Naval Research into funding a VAX 750 at about $75,000, and around that machine built the department's Statistical Computing Facility, of which he was founding director [6][7].

Breiman retired in 1993 but continued as a Professor in the Graduate School, holding National Science Foundation grants and supervising three more doctoral students [7]. Bickel later noted that some of his best work, random forests included, came after retirement [7]. The Mathematics Genealogy Project records eight doctoral students in all, among them Adele Cutler (Berkeley, 1988), who would become his collaborator on random forests [8].

## CART

*Classification and Regression Trees*, written with Jerome Friedman, Richard Olshen and Charles Stone, appeared in 1984 from Wadsworth and has been kept in print by Chapman & Hall and CRC Press ever since [9]. Breiman called it the most significant thing he did after leaving UCLA, and recalled that once the book was out its authors were so fed up with thinking about trees that their interests turned elsewhere [6].

CART grows binary trees by recursive partitioning: at each node the algorithm searches over variables and split points for the split that most reduces node impurity, using [Gini impurity](https://aiwiki.ai/wiki/gini_impurity) for classification and squared error for regression [10][11]. Two ideas in the book proved as important as the tree-growing itself. The first is cost-complexity pruning, the book's answer to [overfitting](https://aiwiki.ai/wiki/overfitting): rather than stopping early, CART grows a large tree and then chooses among nested subtrees by minimizing risk plus a complexity parameter times the number of terminal nodes, with the parameter selected by [cross-validation](https://aiwiki.ai/wiki/cross-validation) [11]. The second is surrogate splits, which handle missing values by re-running the partitioning search to find other variables that reproduce the chosen split, then routing cases with missing data down the best available surrogate [11]. The book's final chapter proves that CART converges asymptotically to the Bayes risk as trees are allowed to grow with sample size [5].

CART differs from the contemporaneous [ID3](https://aiwiki.ai/wiki/id3_algorithm) and [C4.5](https://aiwiki.ai/wiki/c4_5) family in supporting numeric targets, in restricting itself to binary splits, and in producing no rule sets [10]. Its influence is easiest to see in software: [scikit-learn](https://aiwiki.ai/wiki/scikit_learn) states plainly that it "uses an optimized version of the CART algorithm," and the R package rpart carries a different acronym in part because CART was the trademarked name of a particular software implementation of the book's ideas [10][11].

## Bagging

Breiman circulated "Bagging Predictors" as Berkeley Technical Report No. 421 in September 1994; it appeared in *Machine Learning* in 1996 [2][12]. The method takes repeated bootstrap samples from the training set, fits a predictor to each, and aggregates: average the predictions for a numeric response, take a plurality vote for a class label. "We call this procedure 'bootstrap aggregating' and use the acronym bagging" [2].

The paper is unusually clear about when the trick works and when it does not. "The vital element is the instability of the prediction method," Breiman wrote. "If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy" [2]. He had shown earlier that [neural networks](https://aiwiki.ai/wiki/neural_network), classification and regression trees, and subset selection in linear regression are unstable in this sense, while k-nearest-neighbor methods are stable; bagging helps the former and can slightly hurt the latter [2]. Reported reductions in test set misclassification for bagged classification trees ranged from 20 to 47 percent, and reductions in test set mean squared error for bagged regression trees from 22 to 46 percent [2].

The idea arrived by way of a different problem. Breiman had been studying instability in subset selection for linear regression, realized that perturbing the data and averaging the resulting best-subset predictors stabilized them, and then asked why the same treatment would not work on trees [6].

## Arcing and the boosting argument

When Yoav Freund and Robert Schapire's AdaBoost turned out to beat bagging on most data sets, Breiman spent several years trying to work out why. He grouped [AdaBoost](https://aiwiki.ai/wiki/adaboost) and its relatives under the name arcing, coined because the basis of such algorithms is to "adaptively resample and combine," and published "Arcing classifier" with discussion in the *Annals of Statistics* in 1998 [6][13]. The paper framed ensemble gains in terms of the [bias-variance tradeoff](https://aiwiki.ai/wiki/bias_variance_tradeoff): unstable classifiers such as trees can have low bias across a wide range of problems, and their real weakness is variance, which both bagging and arcing reduce [13]. His 2000 technical report "Some infinity theory for tree ensembles" carried a section headed "My kingdom for some good theory," a fair summary of how satisfied he was with the state of the explanation [5].

## Random forests

"Random Forests" was written in January 2001 and published in *Machine Learning* later that year [3][4]. Breiman's Definition 1.1 is deliberately broad: "A random forest is a classifier consisting of a collection of tree-structured classifiers {h(x,Θk), k=1,...} where the {Θk} are independent identically distributed random vectors and each tree casts a unit vote for the most popular class at input x" [3]. Bagging, random split selection, and the random subspace method of Ho all fall inside that definition; the specific recipe the paper advocates, and the one the name now denotes, grows each tree on a bootstrap sample and, at every node, searches for the best split among m variables drawn at random out of M, with no pruning [3][14].

Two theoretical results anchor the paper. The strong law of large numbers implies that generalization error converges almost surely as trees are added, which as Breiman put it "explains why random forests do not overfit as more trees are added, but produce a limiting value of the generalization error" [3]. And the generalization error is bounded above by the mean correlation between trees times (1 minus the square of their strength) divided by the square of their strength, so accuracy depends on individual trees being strong and on the trees disagreeing with each other [3]. Lowering m reduces both correlation and strength; the optimum sits in a range that is usually wide, which is why the method needs so little tuning [14].

The engineering around those results is what made random forests ubiquitous. Because each bootstrap sample leaves out about a third of the cases, those [out-of-bag](https://aiwiki.ai/wiki/out-of-bag_evaluation_oob_evaluation) cases give an internal, essentially unbiased error estimate with no separate test set or cross-validation loop [14]. Running the out-of-bag cases down each tree with the values of one variable randomly permuted, and measuring how much accuracy drops, yields [permutation variable importance](https://aiwiki.ai/wiki/permutation_variable_importances); summing Gini decreases per variable gives a cheaper alternative [14]. Tree co-occurrence in terminal nodes defines a proximity matrix used for clustering, outlier detection and missing-value imputation [14]. Breiman and Cutler registered Random Forests as a trademark and licensed it exclusively to Salford Systems for commercial software, while the reference Fortran code stayed free; Andy Liaw and Matthew Wiener's R port of that code, still maintained on CRAN, is titled "Breiman and Cutlers Random Forests for Classification and Regression" [14][15].

## Statistical Modeling: The Two Cultures

The essay opens with a picture of nature as a black box turning inputs x into responses y, and then splits statisticians into two populations by what they do about it [5]. The data modeling culture assumes a stochastic model inside the box (linear regression, logistic regression, Cox model), estimates its parameters, and validates by goodness-of-fit tests and residuals. Breiman put its share of the profession at 98 percent. The algorithmic modeling culture treats the interior as unknown, looks for any function that predicts well (decision trees, neural nets), and validates by predictive accuracy on held-out data. He put that at "2% of statisticians, many in other fields" [5].

His charge was that the near-exclusive commitment to data models "has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems" [5]. Three named phenomena carry the argument:

| Name | Claim |
|------|-------|
| Rashomon | "there is often a multitude of different descriptions [equations f(x)] in a class of functions giving about the same minimum error rate," so the variables a fitted model declares important are not stable [5] |
| Occam | "Accuracy generally requires more complex prediction methods. Simple and interpretable functions do not make the most accurate predictors" [5] |
| Bellman | High dimension had long been treated as a curse to be escaped by variable deletion, but adding features can help; shape recognition forests and [support vector machines](https://aiwiki.ai/wiki/support_vector_machine_svm) both win by raising dimensionality [5] |

Breiman rated trees "an A+ on interpretability" and only "a B on prediction," and forests "A+ predictors" that rate an F on interpretability [5]. His resolution was that the choice is framed wrongly: "The goal is not [interpretability](https://aiwiki.ai/wiki/interpretability), but accurate information," and a forest's variable importances and proximities extract more reliable information about the underlying mechanism than a poorly fitting data model does [5]. He illustrated this on a 155-patient hepatitis data set where logistic regression gave a 17.4 percent error rate against 12.3 percent for random forests, and where the two methods disagreed about which covariates mattered [5].

The paper ran with comments from D. R. Cox, Brad Efron, Bruce Hoadley and Emanuel Parzen, plus a rejoinder [5]. Cox accepted that Breiman's target was real but called the portrait of mainstream statistics "a caricature," adding that "like all good caricatures, it contains enough truth and exposes enough weaknesses to be thought-provoking" [5]. Efron objected that "Leo's paper overstates both its role and our profession's lack of interest in it," and that the prediction culture "at least around Stanford, is a lot bigger than 2%" [5]. Breiman's rejoinder was pragmatic rather than triumphal: "Given a statistical problem, find a good solution, whether it is a data model, an algorithmic model or (although it is somewhat against my grain), a Bayesian data model or a completely different approach" [5].

Breiman had form as a critic of applied statistical practice. He spent years auditing the proposed statistical adjustment to the 1990 U.S. Census, concluded from thousands of pages of Census Bureau evaluation documents that well over 80 percent of the estimated adjustment was an artifact of bad data rather than of undercount, published the analysis in *Statistical Science* in 1994, and testified before a Congressional committee on the census in 1997 [5][6].

## Selected works

| Year | Work | Details |
|------|------|---------|
| 1968 | *Probability* | Graduate textbook; SIAM later sought to reissue it in its classics series [1][6] |
| 1984 | *Classification and Regression Trees*, with Jerome Friedman, Richard Olshen and Charles Stone | Wadsworth; later Chapman & Hall and CRC Press [9] |
| 1985 | ACE (alternating conditional expectations), with Friedman | A method for finding nonlinear transformations of predictors and response in regression; named, per Breiman, after an Ace Hardware sign across the street from the bar where he and Friedman were arguing about acronyms [6] |
| 1994 | Analysis of the 1990 Census adjustment | *Statistical Science* [5][6] |
| 1994, 1996 | "Bagging Predictors" | Berkeley Technical Report No. 421; *Machine Learning* 24(2):123-140 [2][12] |
| 1998 | "Arcing classifier (with discussion and a rejoinder by the author)" | *Annals of Statistics* 26(3):801-849 [13] |
| 2001 | "Random Forests" | *Machine Learning* 45(1):5-32 [3][4] |
| 2001 | "Statistical Modeling: The Two Cultures" | *Statistical Science* 16(3):199-231 [5] |

## Honors and death

Breiman was elected to the National Academy of Sciences and the American Academy of Arts and Sciences, was a Fellow of the Institute of Mathematical Statistics and of the American Statistical Association, and received the Berkeley Citation [1][6][7]. He delivered the 2002 Wald Lectures at the 277th meeting of the Institute of Mathematical Statistics in Banff, under the titles "Machine Learning," "Looking Inside the Black Box" and "Software for the Masses" [1]. In June 2005 he received the SIGKDD Data Mining and Knowledge Discovery Innovation Award [7].

He died of cancer at his home in Berkeley on July 5, 2005, at 77 [7]. He had designed that house himself as an amateur architect, and was an accomplished sculptor [6][7]. Peter Bickel, then chairing the Berkeley statistics department, said: "It is trite to say so, but Leo Breiman was indeed a Renaissance man, and we shall miss him greatly" [7].

## Influence and later assessments

The three methods carry the influence. CART is the direct ancestor of the tree learners in scikit-learn and in R's rpart, both of which describe themselves in those terms [10][11]. Bagging and arcing supplied vocabulary that [ensemble learning](https://aiwiki.ai/wiki/ensemble_learning) still uses; the Two Cultures paper already listed bagging, boosting, arcing and additive logistic regression as the familiar methods for growing a [decision forest](https://aiwiki.ai/wiki/decision_forest) [5], the family to which [gradient boosting](https://aiwiki.ai/wiki/gradient_boosting) libraries such as [XGBoost](https://aiwiki.ai/wiki/xgboost) also belong. Random forests and their variants, including [Extra Trees](https://aiwiki.ai/wiki/extra_trees), remain default baselines for [tabular](https://aiwiki.ai/wiki/tabular_models) prediction [17]; the 2001 paper had accumulated more than 120,000 citations in Semantic Scholar's index as of July 2026 [16].

Breiman's substantive claim, that algorithmic models beat data models on messy high-dimensional problems, still holds on the kind of tabular problems he spent his consulting years on. A 2022 benchmark across 45 tabular data sets by Grinsztajn, Oyallon and Varoquaux concluded that "tree-based models remain state-of-the-art on medium-sized data (~10K samples) even without accounting for their superior speed," attributing the gap to inductive biases that neural networks lack: robustness to uninformative features, preservation of the orientation of the data (rotation-invariant networks discard that structure), and the ability to learn irregular functions easily [17].

The essay itself has become a recurring object of argument. In 2021 the journal *Observational Studies* ran a special issue of commentaries on it twenty years after publication, with contributions revisiting the Rashomon effect and the boundary between the two cultures [18]. In July 2025 Cynthia Rudin published "Leo Breiman, the Rashomon Effect, and the Occam Dilemma," which keeps the Rashomon effect but retires the Occam dilemma: after 25 years of powerful computers, she writes, Breiman's claimed accuracy-simplicity tradeoff "is not generally true, in that algorithmic models do not need to be complex to be accurate." The Rashomon effect is her key tool for nullifying that dilemma, and she adds that by "simple" Breiman appears to have considered only linear models or unoptimized decision trees [19]. Breiman himself had written that framing the question as a choice between accuracy and interpretability "is an incorrect interpretation of what the goal of a statistical analysis is" [5].

## See also

- [Random forest](https://aiwiki.ai/wiki/random_forest)
- [CART (Classification and Regression Trees)](https://aiwiki.ai/wiki/cart)
- [Decision tree](https://aiwiki.ai/wiki/decision_tree)
- [Bagging](https://aiwiki.ai/wiki/bagging)
- [Ensemble learning](https://aiwiki.ai/wiki/ensemble_learning)
- [Boosting](https://aiwiki.ai/wiki/boosting)

## References

1. Leo Breiman, personal home page, Department of Statistics, University of California, Berkeley. https://www.stat.berkeley.edu/~breiman/
2. Leo Breiman, "Bagging Predictors," Technical Report No. 421, Department of Statistics, University of California, Berkeley, September 1994. https://www.stat.berkeley.edu/~breiman/bagging.pdf
3. Leo Breiman, "Random Forests," Statistics Department, University of California, Berkeley, January 2001 (author's manuscript). https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf
4. Leo Breiman, "Random Forests," *Machine Learning* 45(1):5-32, October 2001. DOI 10.1023/A:1010933404324. https://doi.org/10.1023/A:1010933404324
5. Leo Breiman, "Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)," *Statistical Science* 16(3):199-231, 2001. DOI 10.1214/ss/1009213726. https://projecteuclid.org/journals/statistical-science/volume-16/issue-3/Statistical-Modeling--The-Two-Cultures-with-comments-and-a/10.1214/ss/1009213726.full
6. Richard Olshen, "A Conversation with Leo Breiman," *Statistical Science* 16(2):184-198, 2001. https://projecteuclid.org/journals/statistical-science/volume-16/issue-2/A-conversation-with-Leo-Breiman/10.1214/ss/1009213290.pdf
7. "Leo Breiman, professor emeritus of statistics, has died at 77," UC Berkeley Press Release, Media Relations, July 7, 2005. https://newsarchive.berkeley.edu/news/media/releases/2005/07/07_breiman.shtml
8. Mathematics Genealogy Project, entry for Leo Breiman (Ph.D., University of California, Berkeley, 1954). https://www.mathgenealogy.org/id.php?id=32157
9. Open Library catalogue records for *Classification and Regression Trees* (Wadsworth International Group, 1984; Chapman & Hall, 1993; CRC Press/Taylor & Francis, 2017). https://openlibrary.org/search?q=Classification+and+Regression+Trees+Breiman
10. scikit-learn user guide, "Decision Trees" (sections on tree algorithms and the CART implementation). https://scikit-learn.org/stable/modules/tree.html
11. Terry M. Therneau and Elizabeth J. Atkinson, "An Introduction to Recursive Partitioning Using the RPART Routines," rpart package vignette, Mayo Foundation. https://cran.r-project.org/web/packages/rpart/vignettes/longintro.pdf
12. Leo Breiman, "Bagging Predictors," *Machine Learning* 24(2):123-140, August 1996. DOI 10.1023/A:1018054314350. https://doi.org/10.1023/A:1018054314350
13. Leo Breiman, "Arcing classifier (with discussion and a rejoinder by the author)," *The Annals of Statistics* 26(3):801-849, June 1998. DOI 10.1214/aos/1024691079. https://projecteuclid.org/journals/annals-of-statistics/volume-26/issue-3/Arcing-classifier-with-discussion-and-a-rejoinder-by-the-author/10.1214/aos/1024691079.full
14. Leo Breiman and Adele Cutler, "Random Forests," project documentation, Department of Statistics, University of California, Berkeley. https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
15. CRAN package randomForest, "Breiman and Cutlers Random Forests for Classification and Regression," version 4.7-1.2, maintained by Andy Liaw. https://cran.r-project.org/web/packages/randomForest/index.html
16. Semantic Scholar API record for Breiman, "Random Forests" (citation count as retrieved July 2026). https://api.semanticscholar.org/graph/v1/paper/DOI:10.1023/A:1010933404324?fields=title,year,citationCount,venue
17. Leo Grinsztajn, Edouard Oyallon and Gael Varoquaux, "Why do tree-based models still outperform deep learning on tabular data?", arXiv:2207.08815, July 2022. https://arxiv.org/abs/2207.08815
18. Alexander D'Amour, "Revisiting Rashomon: A Comment on 'The Two Cultures'," *Observational Studies* 7(1):59-63, 2021 (special issue of commentaries on Breiman's paper); preprint arXiv:2104.02150, April 2021. https://arxiv.org/abs/2104.02150
19. Cynthia Rudin, "Leo Breiman, the Rashomon Effect, and the Occam Dilemma," arXiv:2507.03884, July 2025. https://arxiv.org/abs/2507.03884