Self-training
Last reviewed
Apr 28, 2026
Sources
30 citations
Review status
Source-backed
Revision
v4 · 6,666 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Apr 28, 2026
Sources
30 citations
Review status
Source-backed
Revision
v4 · 6,666 words
Add missing citations, update stale details, or suggest a clearer explanation.
See also: Machine learning terms
Self-training is a semi-supervised learning procedure in which a model trained on a small labeled set is used to generate predictions on unlabeled data, then retrained on its own confident predictions as if they were ground-truth labels. The procedure is also called self-labeling, self-teaching, or bootstrapping, and it is one of the oldest ideas in machine learning that uses unlabeled data: the basic loop appears in H. J. Scudder's 1965 IEEE paper on adaptive pattern recognition machines [1] and was popularized for natural language processing by David Yarowsky's 1995 word sense disambiguation paper [2]. The same loop sits at the heart of modern deep learning techniques such as Pseudo-Label [3], Noisy Student [4], FixMatch [5], and SimCLRv2 [6], and it appears again in large language model post-training pipelines such as Anthropic's Constitutional AI [7] and Stanford's STaR (Self-Taught Reasoner) [8].
The appeal of self-training is its simplicity. The algorithm is essentially "train, predict on unlabeled data, retrain on the predictions, repeat," and it is wrapper-style: any supervised learner can be plugged into the loop. The risk is equally simple to describe. If the initial model is wrong about a pseudo-label and that pseudo-label gets reinforced in the next round, the error compounds. This failure mode is often called confirmation bias or a negative feedback loop, and a substantial fraction of the modern self-training literature (FixMatch, Mean Teacher, Noisy Student, Curriculum Labeling) consists of techniques to keep this loop from going off the rails.
This article describes the basic algorithm, its history, the major variants in vision and NLP, the theoretical understanding (focusing on Wei, Shen, Chen, and Ma's ICLR 2021 analysis [9]), self-distillation as a special case, the recent application of self-training to LLMs, the main failure modes, and how self-training compares to related semi-supervised methods such as co-training and tri-training.
Self-training is a wrapper procedure that turns a supervised learner into a semi-supervised one. Given a labeled set L = {(x_i, y_i)} and an unlabeled set U = {x_j}, self-training repeats the following steps:
The set of pseudo-labels added in step 4 may be removed and recomputed each round (the standard formulation) or accumulated across rounds. The retraining in step 5 may be from scratch or by continuing from the previous parameters. The model in step 5 may be the same architecture as in step 1 or a larger one (the Noisy Student variant).
Self-training is distinguished from the broader family of semi-supervised learning methods (which also includes co-training, graph-based label propagation, generative methods, and consistency regularization) by its wrapper-style structure: it does not require the labeled and unlabeled losses to share a common functional form, and it does not require multiple views of the data. It is closely related to but distinct from transductive learning, in which the model is asked only to label a fixed unlabeled set rather than to generalize to new data.
The canonical self-training loop, as described by Yarowsky [2] and formalized for deep neural networks by Lee [3], takes the form below.
function SelfTraining(L, U, threshold tau, max_iters T):
f = TrainSupervised(L)
for t in 1..T:
Y_hat = f.predict_proba(U)
confident = { (x, argmax_c Y_hat(x, c))
for x in U if max_c Y_hat(x, c) >= tau }
L = L union confident
U = U minus { x : (x, _) in confident }
f = TrainSupervised(L)
if stopping_criterion(f): break
return f
A few choices in this skeleton determine the behaviour:
| Choice | Common values | Effect |
|---|---|---|
| Confidence threshold tau | 0.7 to 0.95 in vision, 0.5 to 1.0 in NLP | Higher thresholds reduce noisy pseudo-labels at the cost of fewer additions per round |
| Selection rule | Top-K most confident, all above threshold, or class-balanced top-K | Class-balanced sampling helps prevent class collapse on imbalanced data |
| Hard vs soft labels | Hard (argmax) labels or soft (full distribution) labels | Soft labels carry uncertainty information and tend to perform better in deep learning |
| Restart vs continue | Train from scratch each round, or continue from previous parameters | Continuing is faster but more vulnerable to confirmation bias |
| Pseudo-label refresh | Recompute every round, or keep accumulated pseudo-labels | Refreshing tracks a moving teacher; accumulating is more stable |
| Stopping criterion | Fixed number of rounds, no improvement on validation, or no new confident pseudo-labels | Fixed-round stopping is most common in deep learning |
This skeleton is what later variants (Pseudo-Label, Noisy Student, FixMatch, SimCLRv2) modify in specific ways. The next section traces those modifications historically.
The earliest reference to a self-training procedure in machine learning is H. J. Scudder's 1965 paper in the IEEE Transactions on Information Theory, "Probability of Error of Some Adaptive Pattern-Recognition Machines" [1]. Scudder analyzed a classifier that uses its own predictions to update its parameters in the absence of teacher labels and derived bounds on the asymptotic probability of error. The work is the standard reference for the observation that self-training can converge to a useful classifier even when the training signal comes entirely from the model's own past decisions, although Scudder also noted that the procedure can fail badly if the initial parameters are far from a good solution. Modern self-training papers usually cite Scudder as the origin point of the technique.
David Yarowsky's 1995 ACL paper, "Unsupervised Word Sense Disambiguation Rivaling Supervised Methods," is the modern reference for self-training in NLP [2]. Yarowsky tackled word sense disambiguation: given a polysemous word such as "plant" (factory or vegetation), decide which sense is intended in a given sentence.
Yarowsky's algorithm exploits two linguistic observations:
The algorithm starts from a small set of seed collocations for each sense (for example, "plant life" for the vegetation sense, "manufacturing plant" for the factory sense). A decision-list classifier is trained on the seed-tagged examples. The classifier is then applied to the rest of the unlabeled corpus, and high-confidence labels are added to the training pool. The process iterates, with the per-discourse constraint applied as a global consistency check at each round.
Yarowsky reported accuracies of 96.5% on a 12-word evaluation, matching or beating fully supervised classifiers trained on hand-tagged corpora. The paper also explicitly used the words "bootstrapping" and "self-training," which are the terms still used today. Subsequent theoretical analysis by Abney [10] and others has shown that the Yarowsky algorithm can be understood as approximately optimizing a log-likelihood under a particular probabilistic model, which gave the procedure a firmer footing.
For a long period after Yarowsky, the consensus in NLP was that self-training did not work for syntactic parsing, because parsers were already strong and pseudo-labels were too noisy. McClosky, Charniak, and Johnson's 2006 NAACL paper, "Effective Self-Training for Parsing," reversed this view by showing that a Charniak parser could be improved by self-training when paired with a separate reranker [11]. The reranker effectively broke the symmetry of the self-training loop (the parser was not training on its own raw predictions but on reranked predictions), which prefigured the multi-model schemes (teacher-student, FixMatch, Mean Teacher) that came to dominate the deep-learning era.
Dong-Hyun Lee's 2013 ICML Workshop paper, "Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks," applied the self-training idea to deep neural networks under the name pseudo-labeling [3]. Lee's procedure is essentially the canonical self-training loop applied to a neural network, with one important detail: the labeled-loss term and the pseudo-labeled-loss term are combined within each minibatch with a time-dependent weighting alpha(t) that ramps up from zero. This avoided the problem of the model learning the wrong thing too early in training, before the supervised classifier was strong enough to produce reliable pseudo-labels.
Lee showed that pseudo-labeling improved MNIST accuracy in low-label regimes and connected the approach to entropy regularization in semi-supervised learning. The paper also established the term "pseudo-label" that is now standard. Pseudo-Label was for several years the simplest deep-learning baseline for semi-supervised image classification.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le's CVPR 2020 paper, "Self-Training with Noisy Student Improves ImageNet Classification," pushed self-training to a state-of-the-art result on the standard benchmark of computer vision [4]. The recipe is:
The final EfficientNet-L2 model reached 88.4% top-1 accuracy on ImageNet, beating the previous state of the art (which used 3.5 billion weakly labeled Instagram images) by 2.0 percentage points and substantially improving robustness on ImageNet-A, ImageNet-C, and ImageNet-P. Two design choices are essential to the result. First, the student is larger than the teacher, so it has more capacity to fit the augmented dataset. Second, the student is trained with strong noise (the teacher is not), which forces the student to learn more invariant features than the teacher. Without these choices, naive self-training plateaus on ImageNet rather than improving.
Noisy Student was the proof point that self-training was not just a low-resource trick but could push the absolute frontier of vision models when paired with enough unlabeled data and enough compute.
Kihyuk Sohn and colleagues' NeurIPS 2020 paper, "FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence," demonstrated that the self-training and consistency regularization families of semi-supervised learning could be unified into a strikingly simple algorithm [5]. FixMatch works as follows:
FixMatch reaches 94.93% accuracy on CIFAR-10 with only 250 labeled images, and 88.61% with only 40 labels (4 per class), competitive with fully supervised training that uses 50,000 labels. The strong-versus-weak augmentation asymmetry is the key trick: the pseudo-label comes from the easy view (which the model is more likely to get right), and the supervised loss is applied to the hard view (which forces invariance to perturbations). This gives FixMatch the noise-injection benefits of Noisy Student in a single-pass training procedure rather than an iterative teacher-student loop.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton's NeurIPS 2020 paper, "Big Self-Supervised Models Are Strong Semi-Supervised Learners," applied a three-stage recipe in which self-training appears in the final stage [6]. The pipeline is:
The result was 73.9% top-1 ImageNet accuracy with only 1% of labels (using ResNet-50 as the student), a tenfold improvement in label efficiency over the previous state of the art. The two main lessons were that bigger pretrained models are more label-efficient, and that the big model can be distilled into a small one with little loss in accuracy. The distillation step is essentially a one-round self-training procedure with a fixed teacher, which puts SimCLRv2 in the same algorithmic family as Pseudo-Label and Noisy Student.
In neural machine translation, self-training takes a specialized form known as back-translation, introduced by Sennrich, Haddow, and Birch in their 2016 ACL paper "Improving Neural Machine Translation Models with Monolingual Data" [12]. Given a translation system that maps source language A to target language B, back-translation works as follows:
Back-translation is self-training with a twist: the pseudo-labels are inputs (synthetic source sentences) rather than outputs (target sentences). The technique gave +2.8 to +3.7 BLEU on WMT 15 English-German and +2.1 to +3.4 BLEU on the low-resource IWSLT 14 Turkish-English benchmark, and it has been a standard ingredient of competitive machine translation systems ever since. Iterative back-translation, in which the forward and reverse models alternate training rounds, is a direct analogue of the iterated teacher-student loop in Noisy Student.
The canonical self-training loop has been extended in many directions. The table below summarizes the most influential variants.
| Variant | Year | Authors | Key idea | Domain |
|---|---|---|---|---|
| Yarowsky algorithm | 1995 | Yarowsky | Decision-list classifier with one-sense-per-collocation and one-sense-per-discourse constraints | NLP (word sense) |
| Self-training for parsing | 2006 | McClosky, Charniak, Johnson | Reranker breaks symmetry of self-training loop | NLP (parsing) |
| Pseudo-Label | 2013 | Lee | Time-ramped weighting on pseudo-label loss for deep nets | Vision |
| Back-translation | 2016 | Sennrich, Haddow, Birch | Reverse model produces synthetic source sentences | Machine translation |
| Mean Teacher | 2017 | Tarvainen, Valpola | Teacher is exponential moving average of student weights | Vision |
| Noisy Student | 2020 | Xie, Luong, Hovy, Le | Larger student, strong noise injection on student, iterated | Vision (ImageNet) |
| FixMatch | 2020 | Sohn et al. | Weak augmentation produces pseudo-label, strong augmentation receives it | Vision |
| SimCLRv2 distillation | 2020 | Chen, Kornblith, Swersky, Norouzi, Hinton | Big self-supervised teacher distilled into small student | Vision |
| Curriculum Labeling | 2021 | Cascante-Bonilla, Tan, Qi, Ordonez | Quantile-based threshold over training rounds | Vision |
| STaR | 2022 | Zelikman, Wu, Mu, Goodman | Self-train an LLM on its own correct chain-of-thought rationales | LLMs |
| Constitutional AI / RLAIF | 2022 | Bai et al. (Anthropic) | LLM critiques and revises its own outputs against a constitution | LLMs |
Antti Tarvainen and Harri Valpola's 2017 NeurIPS paper, "Mean Teachers Are Better Role Models," introduced a particularly influential variant in which the teacher is not a separate model but the exponential moving average (EMA) of the student's own weights over time [13]. The student is trained with the usual supervised loss on labeled data plus a consistency loss that pushes the student's predictions on unlabeled data toward the EMA-teacher's predictions. Mean Teacher is technically a consistency-regularization method rather than a strict pseudo-labeling method, but it sits in the same algorithmic neighbourhood: the EMA-teacher's predictions function as soft, slowly evolving pseudo-labels. Mean Teacher reached 4.35% error on SVHN with only 250 labels and was for several years the strongest semi-supervised baseline before FixMatch.
A recurring theme in modern self-training is that the confidence threshold should not be fixed throughout training. Curriculum Labeling (Cascante-Bonilla, Tan, Qi, Ordonez, AAAI 2021) [14] uses a quantile-based threshold so that, for example, the top 20% of pseudo-labels are added in round 1, the top 40% in round 2, and so on. This curriculum-style schedule mirrors the Yarowsky algorithm's tendency to start with the most certain examples and expand to harder ones over time. UPS (Uncertainty-aware Pseudo-Label Selection) [15] adds Monte Carlo dropout to estimate epistemic uncertainty and selects pseudo-labels with both high probability and low uncertainty.
Self-training is one of several major families of semi-supervised learning. The table below contrasts them.
| Method | Mechanism | Requires multiple views? | Requires unlabeled loss? | Failure mode |
|---|---|---|---|---|
| Self-training (pseudo-labeling) | Train, predict, retrain on confident predictions | No | No (uses standard supervised loss on pseudo-labels) | Confirmation bias on noisy pseudo-labels |
| Co-training (Blum and Mitchell 1998) [16] | Two models on two conditionally independent views label data for each other | Yes (two sufficient views) | No | Breaks down when views are not conditionally independent |
| Tri-training (Zhou and Li 2005) [17] | Three models; an unlabeled example is labeled when two agree | No (uses bootstrap samples instead of views) | No | Less label noise than self-training but more computation |
| Consistency regularization (Pi-Model, Mean Teacher, FixMatch) | Penalize differences between predictions on perturbed copies of the same input | No | Yes (consistency loss) | Sensitive to choice of perturbation |
| Generative models (mixture models, deep generative SSL) | Model joint distribution p(x, y) using unlabeled data | No | Yes (likelihood term) | Model misspecification can hurt |
| Graph-based label propagation | Spread labels through a similarity graph | No (graph encodes structure) | Yes (smoothness loss) | Requires meaningful similarity metric |
| Entropy minimization | Add a low-entropy preference on unlabeled predictions | No | Yes (entropy term) | Encourages overconfident predictions |
| Self-supervised pretraining | Pretrain on a pretext task without labels, then fine-tune | No | Yes (pretext loss) | Pretext task may not transfer |
Many modern systems combine families. FixMatch combines self-training (pseudo-labeling on weak augmentation) with consistency regularization (matching prediction on strong augmentation). SimCLRv2 combines self-supervised pretraining with self-training distillation. The boundary between "pure self-training" and "hybrid semi-supervised method" is blurry in practice.
A persistent challenge for self-training has been that it lacked a theoretical justification beyond linear models for a long time. Why should retraining on a model's own predictions improve the model? Naively, the new pseudo-labels carry no information that was not already in the original predictions, so the gradient updates should average to zero.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma's ICLR 2021 oral paper, "Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data," gave the first theoretical analysis of self-training that applies to deep neural networks [9]. Their core idea is that self-training works when there is an expansion property: data points that are close to each other (in the sense of a small input perturbation) tend to belong to the same class. Under this assumption, the consistency loss imposed by self-training ("the prediction on a perturbed copy of an input should match the original prediction") propagates labels from the labeled set outward through neighborhoods, eventually covering the entire data manifold.
Wei et al. showed that this analysis applies to neural networks, not just linear models, because the loss landscape is no longer the bottleneck once the expansion assumption is in place. Their result also covers consistency regularization (FixMatch, Mean Teacher) and unsupervised domain adaptation as special cases, providing a unified theoretical framework. The paper was selected as an oral presentation at ICLR 2021 and is now the standard theoretical reference for why deep self-training works.
The Wei et al. analysis is consistent with the long-standing folk wisdom in self-training: the labeled examples must be sufficiently representative of the underlying classes that nearby unlabeled examples are correctly classified by the initial model. When this holds, the loop expands the labeled neighbourhoods. When it does not, the loop drifts.
Before the deep learning era, theoretical analysis of self-training was restricted to linear models, decision lists, and mixture models. Abney's 2004 paper, "Understanding the Yarowsky Algorithm," reformulated the Yarowsky procedure as approximate optimization of a log-likelihood [10] and clarified what assumptions it implicitly relied on. Haffari and Sarkar's analyses [18] gave conditions under which self-training converges. The literature on the related EM algorithm provided additional results because EM with unlabeled data can be viewed as a soft version of self-training.
More recent analyses have examined self-training in high-dimensional Gaussian mixture settings, showing that pseudo-labeling can either help or hurt depending on the signal-to-noise ratio and the relative sizes of the labeled and unlabeled sets [19]. The general finding is that self-training is most beneficial when the initial model is already reasonably good (so pseudo-labels are mostly correct) and when the unlabeled set is large enough to substantially expand the effective training distribution. When the initial model is weak, self-training can amplify rather than correct its errors.
Self-distillation is the special case of self-training in which the teacher and student have the same architecture (or even identical model classes). The idea was given a name and a careful empirical study by Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar in their 2018 ICML paper, "Born-Again Neural Networks" [20]. Their procedure trains a teacher in the usual way, then trains a student of the same architecture using the teacher's outputs as soft targets, then trains a third generation using the second's outputs, and so on. Surprisingly, each generation slightly outperforms the previous one on CIFAR-10 and CIFAR-100, with the BAN-DenseNets reaching 3.5% error on CIFAR-10 and 15.5% error on CIFAR-100.
Self-distillation can be viewed through several lenses:
Self-distillation is widely used in practice. It is one of the components of SimCLRv2 (the supervised fine-tuned big model is distilled into the smaller deployment model) and of Noisy Student (later iterations are essentially self-distillation with noise). It is also used heavily in knowledge distillation pipelines for compressing large models into smaller deployable ones.
The most active area of self-training research in 2025 and 2026 is the post-training of large language models. Several distinct lines of work apply the self-training pattern in different ways.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman's NeurIPS 2022 paper, "STaR: Bootstrapping Reasoning with Reasoning," applied self-training to chain-of-thought reasoning [8]. The loop is:
STaR showed that this loop dramatically improves reasoning accuracy on math word problems and CommonsenseQA, with a 10x smaller model approaching the accuracy of much larger models. The STaR pattern has since been generalized into many "self-improvement" pipelines for LLMs, including ReST (Reinforced Self-Training), V-STaR, and various rejection-sampling fine-tuning procedures used in the post-training of frontier models.
Anthropic's 2022 paper, "Constitutional AI: Harmlessness from AI Feedback" by Yuntao Bai and colleagues, applied self-training to alignment [7]. The supervised stage works as follows:
This is a form of self-training in which the pseudo-labels are not class predictions but improved completions, and the "correctness" signal comes from the model's own application of the constitution. The reinforcement-learning stage, RLAIF (Reinforcement Learning from AI Feedback), then trains a preference model on AI-generated comparisons and uses it as a reward signal in PPO-style fine-tuning, again replacing the human labels in RLHF with AI-generated ones. Constitutional AI was the proof of concept that AI feedback could substitute for substantial fractions of human feedback in the alignment pipeline. RLAIF has since become a standard ingredient in frontier model post-training, including in subsequent Claude models.
A broader pattern in modern LLM post-training is rejection sampling fine-tuning (RFT), which is essentially self-training applied to instruction following. The loop is:
This procedure has been used in the post-training of LLaMA, Qwen, DeepSeek, and other open and closed frontier models. It is essentially the STaR loop applied to general instruction following rather than to chain-of-thought reasoning specifically. The success of these methods has reopened the question of how much improvement is available from self-distillation alone, without new external data.
The central failure mode of self-training is confirmation bias. Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness analyzed this carefully in their 2020 IJCNN paper, "Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning" [21]. They showed that naive pseudo-labeling is prone to a vicious cycle: incorrect pseudo-labels reinforce themselves, the model becomes increasingly confident in its mistakes, and performance degrades sharply, particularly under class imbalance where minority classes can be erased entirely. Their proposed mitigations include mixup augmentation, ensuring a minimum number of labeled samples per minibatch, and oversampling underrepresented classes.
A related set of failure modes is summarized below.
| Failure mode | Mechanism | Common mitigation |
|---|---|---|
| Confirmation bias | Wrong pseudo-labels are reinforced into harder errors | Strong noise injection, confidence thresholding, mixup |
| Class collapse | Majority classes capture all pseudo-labels; minority classes vanish | Class-balanced selection, per-class thresholds (FlexMatch) |
| Overconfident calibration | Modern deep nets are systematically overconfident; thresholds become meaningless | Temperature scaling, MC dropout, ensemble teachers |
| Domain shift | Unlabeled data has a different distribution from labeled data | Domain-adversarial training, importance weighting |
| Premature commitment | Hard pseudo-labels lock in early errors | Soft pseudo-labels, time-ramped loss weighting (Pseudo-Label) |
| Reward hacking (in LLM self-training) | Model learns to game the judge or verifier rather than improve true performance | Process supervision, multiple judges, holdout evaluation |
| Mode collapse (in synthetic data loops) | Repeated self-training on synthetic data narrows the output distribution | Curate human data into the loop, control diversity, test on held-out distributions |
The Shumailov et al. 2024 Nature paper on "model collapse" [22] generalized confirmation bias to the LLM-on-LLM setting and showed that models trained recursively on their own outputs eventually lose tail behaviour and converge to a narrower distribution. This work is sometimes called model autophagy disorder (MAD) in subsequent literature [23] and is the LLM-scale analogue of the confirmation bias that Arazo et al. identified at the image classification scale.
The self-training literature, taken as a whole, suggests several rules of thumb:
Self-training appears in many production systems and research benchmarks. A non-exhaustive list:
| Domain | Example | System |
|---|---|---|
| Word sense disambiguation | Yarowsky algorithm and successors | NLP toolkits |
| Image classification | Noisy Student, FixMatch, SimCLRv2 | EfficientNet-L2 (state-of-the-art ImageNet at release) |
| Object detection | Self-training detectors (STAC, Soft Teacher) | Modern COCO models |
| Semantic segmentation | Pseudo-label-based SSL for dense prediction | DeepLab variants |
| Speech recognition | Pseudo-labeling and noisy student for ASR | wav2vec 2.0 plus self-training |
| Machine translation | Back-translation, iterative back-translation | Most competitive WMT systems |
| Parsing | McClosky-Charniak-Johnson reranker self-training | Charniak parser |
| Recommendation | Pseudo-labeled implicit feedback | Industrial recommenders |
| LLM reasoning | STaR, V-STaR, ReST | Stanford and DeepMind systems |
| LLM alignment | Constitutional AI, RLAIF | Claude and similar assistants |
| Code generation | RFT on verified completions | DeepSeek-Coder and similar |
| Robotics | Self-training on simulated trajectories | Sim-to-real pipelines |
The pattern of "use the model to generate training data for itself, then verify, then retrain" appears across all of these domains. The verification step (a confidence threshold, a separate verifier, a constitution, an automatic test) is what distinguishes useful self-training from runaway confirmation bias.
Self-training is closely connected to a number of adjacent ideas in machine learning.
In 2026, self-training is no longer a single technique but a design pattern that appears in nearly every part of modern machine learning. The Noisy Student, FixMatch, and SimCLRv2 papers established that self-training can drive state-of-the-art results in vision when paired with sufficient unlabeled data and noise. The STaR and Constitutional AI papers extended the same pattern to LLM reasoning and alignment. The various rejection-sampling fine-tuning recipes used in frontier model post-training are essentially self-training loops with verification gates.
The theoretical work by Wei, Shen, Chen, and Ma [9] gave self-training a respectable footing in the deep-learning era, and the practical work by Arazo et al. [21] and Shumailov et al. [22] mapped out the failure modes carefully enough that practitioners can reason about them in advance. The result is that self-training has moved from a heuristic that sometimes works to a design pattern that is well understood and widely used, with known limitations and known mitigations.
Where self-training is going next is unclear. The most active questions are whether iterated self-training on synthetic data can sustain progress without external data input (the model collapse literature suggests probably not without curation), how to combine self-training with verifiable reward signals for chain-of-thought training, and how to scale RLAIF-style self-improvement to harder reasoning and agentic tasks. All three questions are central to current LLM research, and self-training in some form is part of every proposed answer.
Imagine you're learning to recognize different types of animals. At first, you only know a few animals (the labeled data), but you see many more animals you don't know (the unlabeled data). In self-training, you first learn from the animals you know, then you start making guesses about the animals you don't know. If you're very sure about some of your guesses, you add them to the animals you know and keep learning. You repeat this until you stop getting better.
The smart trick is to add only the guesses you're really, really sure about. The dangerous part is that if you guess wrong on some animals and add those wrong guesses to your list, you might keep getting more confident in your wrong answers. That's called confirmation bias, and it's the main reason this kind of learning sometimes goes badly.
A more recent twist is asking the model to guess and then check itself. In language models, this looks like asking the AI to write out its reasoning, only keeping the answers where the reasoning leads to the right final answer, and then training on those good reasoning examples. That's how systems like STaR and Constitutional AI work: the AI helps train itself, but with a check that prevents it from learning the wrong things.