# Data leakage (machine learning)

> Source: https://aiwiki.ai/wiki/data_leakage
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: Data Science, Machine Learning, Model Evaluation
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Data leakage (machine learning)." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/data_leakage
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Data leakage** (also called **leakage**) in [machine learning](https://aiwiki.ai/wiki/machine_learning) is the use, while a model is being built or evaluated, of information about the prediction target that would not legitimately be available when the model is actually used to make predictions [1][4]. The leaked information can come from a feature that is recorded only after the outcome is known, from a [test set](https://aiwiki.ai/wiki/test_set) that influenced training or [preprocessing](https://aiwiki.ai/wiki/preprocessing), or from evaluation data that is not independent of the training data [3][6]. Its usual effect is an overly optimistic estimate of performance: a leaky model can look accurate in [cross-validation](https://aiwiki.ai/wiki/cross-validation) or on a held-out test set and then do much worse on genuinely new data, for example in production [4].

Leakage has been described as "one of the top ten data mining mistakes" [1], and it is a recurring cause of failed reproductions in research that relies on machine learning. A survey by Sayash Kapoor and Arvind Narayanan of Princeton University found leakage documented in 17 scientific fields and affecting 294 papers [3], and their running tally had grown to 648 papers across 30 fields by May 2024 [13]. A related problem in [large language model](https://aiwiki.ai/wiki/large_language_model) evaluation, where benchmark test data ends up in a model's training data, is usually called benchmark [data contamination](https://aiwiki.ai/wiki/data_contamination) [18].

## Definitions and terminology

Older data mining texts described the problem under other names. Dorian Pyle called the phenomenon anachronisms (something out of place in time) and treated "too good to be true" performance as a giveaway of its presence, while a handbook by Nisbet, Elder and Miner called the issue "leaks from the future" [1]. Saharon Rosset and colleagues defined leakage informally as "the unintentional introduction of predictive information about the target by the data collection, aggregation and preparation process" [1].

A widely cited formal treatment is "Leakage in data mining: Formulation, detection, and avoidance" by Shachar Kaufman, Saharon Rosset and Claudia Perlich, presented at the ACM SIGKDD conference in 2011 [1] and published in an expanded version, with Ori Stitelman as an added author, in ACM Transactions on Knowledge Discovery from Data in December 2012 [2]. The paper defines leakage in terms of legitimacy: every modeling problem has rules for what information the client could actually observe when a prediction is needed, and a model leaks when one of its inputs breaks those rules [1]. The most common rule is what the authors call the "no-time-machine" requirement, under which a legitimate feature may use only information from a time earlier than the target it is predicting [1]. Kaufman and colleagues identified two routes by which illegitimate information enters a model: leaking features, and leaking training examples, the latter arising when training examples are not independent of the targets being evaluated, as in non-stationary problems [1].

Kapoor and Narayanan later defined leakage as a spurious relationship between the input variables and the target that is an artifact of how the data were collected, sampled or preprocessed, one that does not hold in the population about which a scientific claim is made [3]. The [scikit-learn](https://aiwiki.ai/wiki/scikit_learn) documentation gives a practitioner's version: leakage occurs when information that would not be available at prediction time is used when building the model [4].

Practitioner teaching often splits the problem in two. A [Kaggle](https://aiwiki.ai/wiki/kaggle) Learn tutorial by Alexis Cook names the two main types **target leakage**, where predictors include data that will not be available when predictions are made, and **train-test contamination**, where validation or test data influences preprocessing or other fitted steps [6].

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/t0k4415MuZE" title="DSI | Leakage and the Reproducibility Crisis in ML-based Science"></iframe>
<figcaption>"DSI | Leakage and the Reproducibility Crisis in ML-based Science", video from Inside Livermore Lab, published November 2, 2022. <a href="https://www.youtube.com/watch?v=t0k4415MuZE">Watch on YouTube</a>.</figcaption>
</figure>

## Types of leakage

Kapoor and Narayanan organized the failures they found into a taxonomy of eight types in three groups [3].

| Group | Type | What goes wrong |
| --- | --- | --- |
| No clean train-test separation | No test set | Performance is measured on the training data itself |
| No clean train-test separation | Preprocessing on training and test data | Steps such as imputation or oversampling are applied to the whole dataset before splitting |
| No clean train-test separation | Feature selection on training and test data | Features are chosen using the full dataset, test data included |
| No clean train-test separation | Duplicates | The same records appear in both the training and test sets |
| Illegitimate features | Illegitimate features | A feature would not be available at prediction time or acts as a proxy for the outcome |
| Test set not from the distribution of interest | Temporal leakage | Test data comes from before the training data, so the model draws on the future |
| Test set not from the distribution of interest | Non-independence | Related samples, such as repeated measurements of one patient, fall on both sides of the split |
| Test set not from the distribution of interest | Sampling bias | The test set does not represent the population the claim is about |

### Target leakage

Target leakage happens when a feature is recorded after, or as a consequence of, the outcome being predicted. The standard textbook examples collected by Kaufman and colleagues include an "account number" feature in a model predicting whether a customer will open a bank account (account numbers are assigned only once the account exists), and an "interviewer name" feature in a mobile phone churn model, which turned out to be predictive because one salesperson was assigned to customers who had already said they would leave [1]. In a retail website study cited in the same paper, a page-view record carried the total length of the user's session, which is only known after the session ends; replacing it with the page's position in the session removed the leak [1].

Leaks can be indirect. Kaufman and colleagues describe an IBM project that identified likely customers partly from keywords on their websites; the web pages had been collected after those companies became customers, so they contained traces of the IBM products they had bought, such as the word "Websphere" [1]. In medicine, Kapoor and Narayanan cite a hypertension study that used anti-hypertensive drug use as a predictor of hypertension, a feature that both would not be available for a new patient and makes the prediction trivial [3]. The Kaggle tutorial gives a similar example: in a pneumonia model, a column recording whether a patient took antibiotics is filled in after the diagnosis, so it predicts the outcome in training data but is useless for patients who have not yet been treated [6].

The same tutorial walks through a credit card application dataset in which a [random forest](https://aiwiki.ai/wiki/random_forest) reached about 98% cross-validated accuracy. Every applicant who was refused a card had zero card expenditure, while only 2% of accepted applicants did, which suggested the expenditure column described spending on the card being applied for. After dropping that column and three related ones, accuracy fell to a level the tutorial expects to hold up at about 80% on new applications [6].

### Train-test contamination

Train-test contamination occurs when data meant for evaluation shapes any fitted part of the pipeline, not only the model's own weights. The scikit-learn documentation states the rule as "never call `fit` on the test data", and notes that even a simple normalization step leaks if its mean is computed over all of the data rather than over the training subset [4].

[Feature selection](https://aiwiki.ai/wiki/feature_selection) performed before the data are split is a well-documented case. In a scikit-learn example with 200 samples, 10,000 features and completely random labels, selecting the 25 best features on the full dataset and then splitting produced a test accuracy of 0.76, while selecting features on the training split alone produced the chance-level 0.5 [4]. Christophe Ambroise and Geoffrey McLachlan made the same point about gene-expression classifiers in 2002: published results claiming near-zero error rates from a handful of genes had chosen those genes on samples later used for testing, or had run cross-validation that did not repeat the gene selection inside each fold, and with the selection bias corrected the cross-validated error on two published datasets was no longer zero [7]. In a 2024 study of brain-connectivity models, feature selection on combined training and test data raised the correlation between predicted and observed attention problems from 0.01, essentially chance, to 0.48 [10].

[Oversampling](https://aiwiki.ai/wiki/oversampling) for [class-imbalanced data](https://aiwiki.ai/wiki/class-imbalanced_dataset) causes the same problem when it is done before splitting, because synthetic minority examples generated from test records end up in the training set. Gilles Vandewiele and colleagues showed that applying [SMOTE](https://aiwiki.ai/wiki/smote) before partitioning randomly labelled data with no signal yielded an AUC of 0.95 on a "held-out" test set, against 0.49 without SMOTE and 0.48 with SMOTE applied after the split. They identified 24 machine learning studies on a public electrohysterogram database for predicting preterm birth, many reporting near-perfect results, and reproduced eleven of them; they could approach the published figures only by oversampling before partitioning [9].

### Leakage through model selection

Information can also leak through decisions rather than data transformations. If [hyperparameters](https://aiwiki.ai/wiki/hyperparameter) are tuned until a test set gives the best score, knowledge of the test set leaks into the model and the reported metric no longer measures generalization [5]. Sudhir Varma and Richard Simon showed in 2006 that the cross-validation error of a classifier whose parameters had been chosen by cross-validation was a substantially biased estimate of its true error: on simulated "null" data with no real difference between classes, the tuned [support vector machine](https://aiwiki.ai/wiki/support_vector_machine_svm) showed an estimated error below 30% on 38% of datasets, yet performed no better than chance on independent test data. A nested procedure, with tuning in an inner loop and error estimation in an outer loop, gave an estimate very close to the independent-test result [8]. Kaufman and colleagues described a related and harder-to-spot form, in which a modeler who has seen illegitimate information about the deployment population makes design choices (features, algorithms or meta-parameters) informed by it, even though no leaked data appears in training [1].

### Duplicates and non-independent samples

Standard random splits assume that examples are independent. When the same or nearly the same item appears on both sides of a split, the test set partly measures memorization. Björn Barz and Joachim Denzler found near-duplicates of training images for 286 [CIFAR-10](https://aiwiki.ai/wiki/cifar_10) test images and 891 CIFAR-100 test images; counting duplicates within the test sets as well, 3.3% and 10% of the two test sets were affected. A ResNet-110 model had error rates of 0% and 2.9% on those duplicated test images, compared with 5.3% and 26.1% on the full test sets [11].

Grouped data raises the same issue at the level of people or sources. The scikit-learn guide gives the example of medical data with several samples per patient: a model flexible enough to learn person-specific features may fail on new patients, and group-aware splitters such as GroupKFold keep each patient's samples entirely in either the training or the test fold [5]. In the 2024 connectome study, repeating 20% of the subjects in the dataset, so that copies of one person could fall on both sides of a split, inflated the correlation for attention problems by 0.28, while leakage between family members had little effect in the primary dataset [10].

### Temporal leakage

For time series, the scikit-learn documentation notes that classical splitters such as KFold and ShuffleSplit assume independent samples, which autocorrelated data violate; shuffling time-ordered data such as news articles tests the model on items artificially similar to training items and inflates the score. Its TimeSeriesSplit splitter always trains on earlier observations and tests on later ones [5]. Kaufman and colleagues pointed to a practical obstacle: databases often overwrite records without timestamping each change, so information recorded after the prediction time can end up in examples that are supposed to describe the past [1]. A study of machine learning on panel data by Augusto Cerqua, Marco Letta and Gabriele Pinto argued that ignoring the cross-sectional and time structure of such data produces leakage that is hard to detect and inflates out-of-sample performance, illustrating the point with income prediction across more than 3,000 US counties from 2000 to 2019 [15].

## Leakage in data mining competitions

Public competitions produced many of the documented cases. In the 2008 KDD Cup on breast cancer detection from mammography data, the patient ID turned out to be highly predictive, apparently because data from different sources, some selected with knowledge of the patients' condition, had been given consecutive ID ranges [1]. In the INFORMS 2010 Data Mining Challenge on predicting stock price movements, about 30 teams exceeded an AUC of 0.9 and the best surpassed 0.99, results that relied on information from the future and on public stock data that could reveal the target stock's identity; the organizers concluded that verifying that future information had not been used was impossible [1]. Winners of the IJCNN 2011 Social Network Challenge identified the anonymized network as Flickr, de-anonymized most of the data, and used edges from the live Flickr network to correctly predict over 60% of the identified edges [1]. Kaufman and colleagues called leakage from such outside sources "external leakage" and argued that it undermines the organizers' real aim of producing models useful in the domain [1].

## Prevalence and effects in research

Kapoor and Narayanan's 2023 paper in *Patterns* drew on 22 papers that had documented leakage in 17 fields, together affecting 294 papers, and in some cases leading to wildly overoptimistic conclusions [3]. To test the effect directly, they examined 12 civil war prediction papers that used a train-test split and shared code and data. Four contained leakage, and those four were exactly the papers claiming that complex machine learning models beat [logistic regression](https://aiwiki.ai/wiki/logistic_regression). Once the errors were corrected, the complex models did not perform substantively better than the older logistic regression models; in one paper the AUC advantage fell from 0.14 to 0.01 [3].

Studies of code show the same pattern. Chenyang Yang and colleagues built a static analysis tool for Jupyter notebooks, found it 92.9% accurate against manual labels, and ran it on public notebooks from GitHub and Kaggle [12].

| Notebook group | Notebooks analyzed | Preprocessing leakage | Overlap leakage | Multi-test leakage | Any leakage |
| --- | --- | --- | --- | --- | --- |
| GitHub, all | 107,603 | 12.3% | 6.5% | 18.5% | 29.6% |
| GitHub, popular | 920 | 4.9% | 5.2% | 15.9% | 20.9% |
| GitHub, tutorials | 1,157 | 3.9% | 2.9% | 11.3% | 16.2% |
| GitHub, assignments | 7,576 | 13.9% | 7.4% | 22.0% | 33.0% |
| Kaggle, top-voted | 312 | 56.1% | n/a | n/a | n/a |
| Kaggle, recent | 358 | 55.8% | n/a | n/a | n/a |

In the paper's terms, overlap leakage means test rows are used for training (including through augmentation or oversampling), and multi-test leakage means the same test data is evaluated repeatedly, so that model choices are made on it [12].

In neuroimaging, Matthew Rosenblatt and colleagues tested five forms of leakage on connectome-based models across four datasets and three phenotypes. Leakage through feature selection and repeated subjects inflated performance drastically, leaky site correction had little effect, and leaky covariate regression actually lowered performance. In small samples of 100 people the effect of leakage varied widely from run to run, which the authors read as making leakage more unpredictable in the small datasets common in the field [10].

Reviews since then have kept finding the problem. A 2025 letter in *Molecular Psychiatry* reported that at least 20 MRI studies (45%) and 5 clinical studies (38%) in a published meta-analysis of depression treatment prediction described procedures consistent with leakage, in which variables that differed significantly between responders and non-responders were then used to predict that same outcome in the whole dataset. Excluding them lowered the pooled log diagnostic odds ratio for MRI models from 2.53 to 2.02 [17]. A systematic review of 48 studies of EEG-based neonatal seizure detection, published in January 2026, rated 16.7% of the papers as being at high risk of leakage and found that the choice of data-partitioning strategy was a strong predictor of that risk [16].

## Detection

Kaufman and colleagues proposed three ways to find leakage when the modeler does not control data collection. First, exploratory data analysis can reveal features whose predictive power is surprising, such as the patient ID in the 2008 KDD Cup. Second, examining the modeling results can expose a suspicious input in the fitted model, as with the "Websphere" keyword, or performance far above what experience or earlier documented results would lead one to expect. Third, early testing in the field shows leakage as a gap between estimated and realized performance. All three depend on domain knowledge [1]. The Kaggle tutorial applies the same reasoning: it treats 98% accuracy as rare enough to justify inspecting the data, then compares the suspicious column between the two classes [6].

Automated checks exist for some forms. The tool of Yang and colleagues tracks how datasets flow through notebook code into the training and evaluation functions of scikit-learn, Keras and PyTorch, and flags preprocessing, overlap and multi-test leakage [12]. The authors left leakage through dependent data, such as time series, outside the tool's scope, because detecting it requires understanding what the data mean [12].

## Prevention

Most guidance reduces to keeping evaluation data out of every fitted step. The scikit-learn documentation recommends splitting the data before any preprocessing, never including test data when calling `fit` or `fit_transform`, and applying the transformations fitted on the training data to the test data with `transform`. It recommends [pipelines](https://aiwiki.ai/wiki/pipeline), which chain preprocessing and the model so that cross-validation and hyperparameter search refit every step on the training fold only [4]. Hyperparameter search itself needs a separate validation set or nested cross-validation [5][8]. For grouped data, splits should be made by group; for time series, by time [5].

Kaufman and colleagues argued that the most reliable prevention happens at data collection. Their proposal is to attach "legitimacy tags", such as precise timestamps, to every observation and then enforce a "learn-predict separation", building training examples only from information that would have been available for each target, and cutting training data off from evaluation data at a fixed point in time [1]. For competitions, they suggested asking entrants to predict events that occur after the submission deadline, which makes the use of future information physically impossible [1].

For scientific work, Kapoor and Narayanan proposed "model info sheets", a template of 21 questions based on their taxonomy that asks authors to justify a clean train-test split, the legitimacy of each feature, and whether the test set matches the population of interest [3]. A follow-up consensus effort by 19 researchers, REFORMS (recommendations for machine-learning-based science), published a checklist of 32 questions with guidelines in *Science Advances* in 2024 [14], and the 2025 *Molecular Psychiatry* letter recommended REFORMS and PROBAST-AI for judging studies before they are pooled in meta-analyses [17].

## Fixing leakage after the fact

Once leakage is found, removing it is often difficult. Kaufman and colleagues observed that where there is one leaking feature there are usually more, and that deleting the obvious ones can make undetected ones more influential [1]. In a retail example, removing a "total purchase in jewelry" field left a "no purchases in any department" pattern that still revealed jewelry buyers, because every customer in the database had made some purchase [1]. In the 2008 KDD Cup, dropping the patient ID would not necessarily have removed the leak if the different sources also left traces in the images, such as differences in the grey levels of different equipment [1]. The authors concluded that, without a learn-predict separation, fully fixing leakage is typically very hard, and that building a model with some leakage left in is often more practical than trying to remove it all [1].

## Relation to data contamination

In [large language model](https://aiwiki.ai/wiki/large_language_model) evaluation, the closest analogue is benchmark data contamination: evaluation benchmark information that a model has absorbed from its training data, which makes its benchmark results unreliable [18]. It has its own detection methods and literature, covered in the data contamination article.

## See also

- [Data contamination](https://aiwiki.ai/wiki/data_contamination)
- [Overfitting](https://aiwiki.ai/wiki/overfitting)
- [Cross-validation](https://aiwiki.ai/wiki/cross-validation)
- [Holdout data](https://aiwiki.ai/wiki/holdout_data)
- [Splitter](https://aiwiki.ai/wiki/splitter)
- [Training set](https://aiwiki.ai/wiki/training_set)
- [Validation set](https://aiwiki.ai/wiki/validation_set)
- [Concept drift](https://aiwiki.ai/wiki/concept_drift)

## References

1. Kaufman, Shachar; Rosset, Saharon; Perlich, Claudia. "Leakage in data mining: Formulation, detection, and avoidance". Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '11), pp. 556-563, August 2011. https://doi.org/10.1145/2020408.2020496
2. Kaufman, Shachar; Rosset, Saharon; Perlich, Claudia; Stitelman, Ori. "Leakage in data mining: Formulation, detection, and avoidance". ACM Transactions on Knowledge Discovery from Data, 6(4), pp. 1-21, December 2012. https://doi.org/10.1145/2382577.2382579
3. Kapoor, Sayash; Narayanan, Arvind. "Leakage and the reproducibility crisis in machine-learning-based science". Patterns, 4(9), 100804, September 2023. https://pmc.ncbi.nlm.nih.gov/articles/PMC10499856/
4. scikit-learn developers. "Common pitfalls and recommended practices". scikit-learn documentation (version 1.9.1), accessed November 2026. https://scikit-learn.org/stable/common_pitfalls.html
5. scikit-learn developers. "Cross-validation: evaluating estimator performance". scikit-learn documentation, accessed November 2026. https://scikit-learn.org/stable/modules/cross_validation.html
6. Cook, Alexis. "Data Leakage". Kaggle Learn, Intermediate Machine Learning course, accessed November 2026. https://www.kaggle.com/code/alexisbcook/data-leakage
7. Ambroise, Christophe; McLachlan, Geoffrey J. "Selection bias in gene extraction on the basis of microarray gene-expression data". Proceedings of the National Academy of Sciences, 99(10), pp. 6562-6566, April 2002. https://doi.org/10.1073/pnas.102102699
8. Varma, Sudhir; Simon, Richard. "Bias in error estimation when using cross-validation for model selection". BMC Bioinformatics, 7, 91, February 2006. https://doi.org/10.1186/1471-2105-7-91
9. Vandewiele, Gilles; Dehaene, Isabelle; Kovács, György; et al. "Overly Optimistic Prediction Results on Imbalanced Data: a Case Study of Flaws and Benefits when Applying Over-sampling". Artificial Intelligence in Medicine, 111, 101987, 2021 (arXiv:2001.06296). https://arxiv.org/abs/2001.06296
10. Rosenblatt, Matthew; Tejavibulya, Link; Jiang, Rongtao; Noble, Stephanie; Scheinost, Dustin. "Data leakage inflates prediction performance in connectome-based machine learning models". Nature Communications, 15, 1829, February 2024. https://doi.org/10.1038/s41467-024-46150-w
11. Barz, Björn; Denzler, Joachim. "Do We Train on Test Data? Purging CIFAR of Near-Duplicates". arXiv:1902.00423, 2019; Journal of Imaging, 6(6), 41, 2020. https://arxiv.org/abs/1902.00423
12. Yang, Chenyang; Brower-Sinning, Rachel A.; Lewis, Grace A.; Kästner, Christian. "Data Leakage in Notebooks: Static Detection and Better Processes". Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE '22), October 2022. https://arxiv.org/abs/2209.03345
13. Kapoor, Sayash; Narayanan, Arvind. "Leakage and the Reproducibility Crisis in ML-based Science". Princeton University, table updated May 2024. https://reproducible.cs.princeton.edu/
14. Kapoor, S.; Cantrell, E. M.; Peng, K.; et al. "REFORMS: Consensus-based Recommendations for Machine-learning-based Science". Science Advances, 10(18), eadk3452, May 2024. https://doi.org/10.1126/sciadv.adk3452
15. Cerqua, Augusto; Letta, Marco; Pinto, Gabriele. "On the (Mis)Use of Machine Learning With Panel Data". Oxford Bulletin of Economics and Statistics, 88(3), pp. 506-518, 2026 (online September 2025; arXiv:2411.09218). https://doi.org/10.1111/obes.70019
16. Tegenaw, Geletaw Sahle; Song, Hailin; Ward, Tomas. "Generalization or mirage? Data leakage and reported performance in neonatal EEG seizure detection models: a systematic review". BioData Mining, 19, 52, January 2026. https://doi.org/10.1186/s13040-025-00516-y
17. van de Mortel, L. A.; van Wingen, G. A. "Data leakage in machine learning studies creep into meta-analytic estimates of predictive performance". Molecular Psychiatry, 30(12), pp. 6070-6071, 2025. https://doi.org/10.1038/s41380-025-03336-y
18. Xu, Cheng; Guan, Shuhao; Greene, Derek; Kechadi, M-Tahar. "Benchmark Data Contamination of Large Language Models: A Survey". arXiv:2406.04244, June 2024. https://arxiv.org/abs/2406.04244

