BYOL (Bootstrap Your Own Latent)
BYOL (Bootstrap Your Own Latent) is a self-supervised learning method for learning image representations without labels, introduced by Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond and nine colleagues at DeepMind (now Google DeepMind) and Imperial College in June 2020, and published at NeurIPS 2020.[1][2] BYOL trains an "online" network to predict, from one augmented view of an image, the output of a "target" network on a different augmented view of the same image; the target network is not trained by gradient descent but is a slowly moving average of the online network's weights.[1] Unlike the contrastive learning methods that led the field at the time, such as SimCLR and MoCo, BYOL uses no negative pairs (views of different images that the model must push apart).[1]
The paper reported 74.3% top-1 accuracy on ImageNet under linear evaluation with a standard ResNet-50 and 79.6% with a larger ResNet-200 (2x), along with better robustness than SimCLR to smaller batch sizes and to removing image augmentations.[1] Why BYOL does not collapse to a trivial constant representation, given that its loss alone would allow one, prompted a series of follow-up studies, among them work on the role of batch normalization, SimSiam, and a theoretical analysis of non-contrastive learning dynamics.[4][5][6][8] Later methods such as DINO cite it as an inspiration, and its bootstrapped-prediction approach was adapted to graphs, audio and reinforcement learning.[9][11][12][13]
Background
By 2020 the strongest self-supervised image methods were contrastive: they pulled together the representations of two augmented views of the same image (a positive pair) and pushed apart the representations of views of different images (negative pairs).[1] The BYOL authors noted that these methods needed careful handling of negatives, through large batches (SimCLR), memory banks (MoCo) or custom mining strategies, and that their performance depended heavily on the choice of image augmentations.[1]
Predicting one view's representation from another view's representation without negatives runs into a known problem: a representation that is constant across all inputs predicts itself perfectly, so the objective admits collapsed solutions.[1] Contrastive methods avoid this by turning prediction into discrimination between the matching view and the negatives.[1] The BYOL paper set out to test whether negatives are indispensable for avoiding collapse while keeping high performance.[1]
The starting observation was an experiment with a fixed, randomly initialized target network. Training a network to predict that frozen network's outputs gave 18.8% top-1 accuracy on ImageNet linear evaluation, against 1.4% for the random network itself.[1] The authors took this as the core motivation: a new representation trained to predict a given target representation can be better than the target, and iterating the procedure could produce a sequence of improving representations.[1] BYOL replaces the sequence of fixed checkpoints with a continuously updated exponential moving average of the online network.[1] The paper states that "bootstrap" is used in its idiomatic sense, not in the statistical sense.[1]
The authors credited deep reinforcement learning for the idea of a slow-moving target network that stabilizes bootstrapped updates, as with the target network used to stabilize Bellman-equation updates.[1] They also related BYOL to Mean Teacher, a semi-supervised method whose "teacher" is a moving average of a "student" network, and to MoCo, whose momentum encoder keeps the representations of negatives consistent; BYOL instead uses its moving-average network to produce prediction targets.[1]
Method
Architecture
BYOL uses two networks with the same backbone architecture but different weights:[1]
| Component | Online network (weights $\theta$) | Target network (weights $\xi$) |
|---|---|---|
| Encoder | $f_\theta$, produces representation $y_\theta$ | $f_\xi$, produces $y'_\xi$ |
| Projector | $g_\theta$, produces projection $z_\theta$ | $g_\xi$, produces $z'_\xi$ |
| Predictor | $q_\theta$, produces prediction $q_\theta(z_\theta)$ | None |
| How weights change | Gradient descent on the loss | Exponential moving average of $\theta$ |
Because the predictor exists only on the online branch, the architecture is asymmetric.[1] After training, everything except the online encoder $f_\theta$ is discarded, and $y_\theta$ is used as the image representation.[1]
Loss
An image $x$ is turned into two views, $v = t(x)$ and $v' = t'(x)$, using augmentations drawn from two distributions $\mathcal{T}$ and $\mathcal{T}'$.[1] The online network processes $v$ and the target network processes $v'$. The prediction $q_\theta(z_\theta)$ and the target projection $z'_\xi$ are each normalized to unit $\ell_2$ norm (written with a bar), and the loss is their mean squared error, which equals 2 minus twice their cosine similarity:[1]
The loss is symmetrized by also feeding $v'$ to the online network and $v$ to the target network, giving $\tilde{\mathcal{L}}{\theta,\xi}$; the total loss is $\mathcal{L}^{\text{BYOL}}{\theta,\xi} = \mathcal{L}{\theta,\xi} + \tilde{\mathcal{L}}{\theta,\xi}$.[1] It is minimized with respect to $\theta$ only. No gradient flows into the target branch (a stop-gradient).[1]
Target update
After each training step the target weights move toward the online weights:[1]
where $\tau \in [0, 1]$ is the target decay rate. In the main experiments $\tau$ starts at $\tau_{\text{base}} = 0.996$ and rises to 1 over training following $\tau = 1 - (1 - \tau_{\text{base}}) \cdot (\cos(\pi k / K) + 1)/2$, with $k$ the current step and $K$ the total number of steps.[1] At $\tau = 1$ the target never changes, and at $\tau = 0$ it is an instant copy of the online network.[1]
Why it does not collapse (the authors' hypothesis)
The paper acknowledged that BYOL's objective admits collapsed solutions, such as outputting the same vector for every image, and reported that it did not observe convergence to them in experiments.[1] The authors offered a hypothesis rather than a proof. Because the target parameters $\xi$ are not updated along the gradient of the loss, they argued, there is probably no single loss that BYOL's dynamics minimize jointly over $\theta$ and $\xi$, a situation they compared to generative adversarial networks.[1] If the predictor is assumed optimal, the online updates follow the gradient of the expected conditional variance of the target projection given the online projection; discarding information from the online projection cannot reduce that variance, so constant features would be unstable.[1] They suggested that the main role of the moving-average target is to keep the predictor near-optimal throughout training.[1]
Two experiments supported this reading. Without a target network, training did not collapse when the predictor was kept near-optimal, either by solving for an optimal linear predictor on each batch (52.5% top-1) or by raising the predictor's learning rate (66.5% top-1).[1]
Implementation
BYOL used the image augmentations from SimCLR, with different probabilities for the two view distributions:[1]
| Augmentation | Probability in $\mathcal{T}$ | Probability in $\mathcal{T}'$ |
|---|---|---|
| Random resized crop (8% to 100% of area), resized to 224x224 | 1.0 | 1.0 |
| Horizontal flip | 0.5 | 0.5 |
| Color jitter (brightness 0.4, contrast 0.4, saturation 0.2, hue 0.1) | 0.8 | 0.8 |
| Grayscale conversion | 0.2 | 0.2 |
| Gaussian blur | 1.0 | 0.1 |
| Solarization | 0.0 | 0.2 |
The base encoder was a ResNet-50 (v1, post-activation) whose 2,048-dimensional average-pooled output is the representation; deeper and wider ResNets were also tested.[1] The projector is a multilayer perceptron with a 4,096-unit hidden layer, batch normalization and ReLU, and a 256-dimensional output; unlike SimCLR, its output is not batch-normalized. The predictor has the same architecture.[1]
| Setting | Value |
|---|---|
| Optimizer | LARS, cosine learning-rate decay without restarts, 10-epoch warm-up |
| Base learning rate | 0.2, scaled linearly with batch size (0.2 x batch size / 256) |
| Weight decay | 1.5e-6, excluding biases and batch-norm parameters |
| Training length | 1,000 epochs |
| Batch size and hardware | 4,096 over 512 Cloud TPU v3 cores |
| Training time | About 8 hours for ResNet-50 |
Source for the table: Grill et al. (2020).[1] With a batch size of 512 on 64 TPU cores, a base learning rate of 0.4 and $\tau_{\text{base}} = 0.9995$, BYOL reached 73.7% top-1 after 1,000 epochs, which took about four days.[1] The paper included JAX and Haiku pseudo-code.[1]
Results
All results below are for representations pre-trained on the ImageNet (ILSVRC-2012) training set without labels.[1]
Linear evaluation on ImageNet
Under the standard linear protocol, a linear classifier is trained on the frozen representation.[1]
| Encoder | Parameters | BYOL top-1 (%) | Comparison reported in the paper |
|---|---|---|---|
| ResNet-50 | 24M | 74.3 (top-5 91.6) | InfoMin Aug. 73.0; MoCo v2 71.1; SimCLR 69.3 |
| ResNet-50 (2x) | 94M | 77.4 | SimCLR 74.2; CMC 70.6 |
| ResNet-50 (4x) | 375M | 78.6 | SimCLR 76.5; MoCo 68.6 |
| ResNet-200 (2x) | 250M | 79.6 | Previous best 76.8 |
Source for the table: Grill et al. (2020).[1] The ResNet-50 result was 1.3 points above the previous self-supervised state of the art and closer to the 76.5% supervised baseline reported by SimCLR, but still below a stronger 78.9% supervised baseline cited in the paper.[1] The authors noted that the ResNet-200 (2x) result improved on the previous best while using 30% fewer parameters.[1]
Semi-supervised learning
Fine-tuned on 1% or 10% of ImageNet's labels, using the same fixed splits as SimCLR, a BYOL ResNet-50 reached 53.2% and 68.8% top-1, against 48.3% and 65.6% for SimCLR and 25.4% and 56.4% for a supervised baseline.[1] Fine-tuned on all labels, the BYOL ResNet-50 reached 77.7%.[1]
Transfer
On 12 classification datasets (including Food-101, CIFAR-10 and CIFAR-100, SUN397, Pets and Flowers), the paper reported that BYOL outperformed SimCLR on all benchmarks (its table shows the authors' SimCLR reproduction marginally ahead in two cells, DTD under linear evaluation and SUN397 after fine-tuning) and beat the supervised ImageNet baseline on 7 of the 12 (under both linear evaluation and fine-tuning), while being slightly worse on the other five.[1] On other vision tasks with a ResNet-50:[1]
| Task | Metric | Supervised ImageNet | SimCLR (reproduction) | BYOL |
|---|---|---|---|---|
| PASCAL VOC 2012 semantic segmentation | mIoU | 74.4 | 75.2 | 76.3 |
| PASCAL VOC 2007 object detection (Faster R-CNN) | AP50 | 74.4 | 75.2 | 77.5 |
| NYU v2 depth estimation | pct. < 1.25 (higher is better) | 81.1 | 83.3 | 84.6 |
| NYU v2 depth estimation | rms error (lower is better) | 0.573 | 0.557 | 0.541 |
Source for the table: Grill et al. (2020).[1] An appendix repeated the evaluation after pre-training on the Places365 scene dataset instead of ImageNet.[1]
Ablations
The ablations ran for 300 epochs, averaged over three seeds, and used $\tau_{\text{base}} = 0.99$; the baseline in this setting was 72.5%.[1]
Batch size. SimCLR's accuracy fell quickly as the batch shrank, which the authors attributed to fewer negatives. BYOL stayed stable from 4,096 down to 256 and dropped only at smaller sizes, which the authors attributed to batch normalization, the only batch-size dependency in their pipeline.[1]
Augmentations. Removing color distortion cost BYOL 9.1 points and SimCLR 22.2 points. With random crops as the only augmentation, BYOL kept 59.4% (13.1 points below its 72.5% baseline), while SimCLR fell to 40.3% (27.6 points below 67.9%).[1] The authors' explanation was that a contrastive task built only on crops can be solved with color histograms, whereas BYOL is pushed to keep any information that the target representation captures.[1]
Target decay rate.[1]
| Target | $\tau_{\text{base}}$ | Top-1 (%) |
|---|---|---|
| Constant random network | 1 | 18.8 |
| Moving average of online network | 0.999 | 69.8 |
| Moving average of online network | 0.99 | 72.5 |
| Moving average of online network | 0.9 | 68.4 |
| Copy of online network (stop-gradient only) | 0 | 0.3 |
Predictor, target network and negatives. The paper wrote SimCLR and BYOL as special cases of one generalized InfoNCE objective with a weight $\beta$ on the negative term.[1] Without negatives ($\beta = 0$), only the combination of a predictor and a target network worked (72.5%); removing either collapsed to 0.3% or less.[1] Removing the predictor gives an unsupervised version of Mean Teacher, and that variant collapsed.[1] The authors reported that simply adding a target network to SimCLR improved it by 1.6 points, and presented this as evidence that a target network helps through stabilization alone, separately from MoCo's use of it to supply more negatives.[1] Removing weight decay made both BYOL and SimCLR diverge.[1]
The batch normalization debate
In a blog post published in August 2020 and now hosted by Imbue, Josh Albrecht and Abe Fetterman described their reproduction of BYOL and reported that it "often performs no better than random when batch normalization is removed."[4] They argued that batch normalization acts as an implicit contrastive signal: because it normalizes each example against the statistics of the mini-batch, the other examples serve as implicit negatives, and "all samples in the mini-batch cannot take on the same value."[4]
The BYOL team answered in October 2020 with "BYOL works even without batch statistics" (Richemond et al.).[5] They confirmed that removing every batch-normalization layer stopped BYOL from learning in the standard setup, but rejected the implicit-negative explanation.[5] With a better initialization and with batch normalization's trainable scale and bias kept, BYOL trained without any normalization reached 65.7%.[5] Replacing batch normalization with group normalization plus weight standardization, neither of which uses batch statistics, reached 73.9% against 74.3% for standard BYOL.[5] The authors hypothesized that batch normalization's main role in BYOL is to compensate for poorly scaled initialization.[5]
Follow-up analysis
SimSiam. In November 2020 Xinlei Chen and Kaiming He of Facebook AI Research described SimSiam, a Siamese network that uses neither negative pairs nor a momentum encoder and works with ordinary batch sizes.[6] They summarized it as "BYOL without the momentum encoder" and reported that a stop-gradient operation, not the momentum encoder, is what prevents collapse in this family; the moving average may still improve accuracy.[6] The paper appeared at CVPR 2021.[7]
Learning dynamics. Yuandong Tian, Xinlei Chen and Surya Ganguli (ICML 2021) analyzed the nonlinear learning dynamics of non-contrastive methods such as BYOL and SimSiam in simple linear networks, to explain how predictor networks, stop-gradients, exponential moving averages and weight decay interact to avoid collapse.[8] From that analysis they derived DirectPred, which sets a linear predictor directly from the statistics of its inputs instead of training it; on ImageNet it performed comparably to a two-layer nonlinear predictor with batch normalization and outperformed a linear predictor by 2.5 points in 300-epoch training.[8]
Vision transformers. In MoCo v3, Chen, Xie and He benchmarked BYOL with vision transformers. Freezing the patch-projection layer at its random initialization raised BYOL's 100-epoch ViT-B/16 accuracy from 69.7% to 71.0%, and in a 300-epoch comparison BYOL reached 71.0% with ViT-S and 73.9% with ViT-B, behind MoCo v3's 72.5% and 76.5%.[14]
Influence and extensions
| Work | Year | Relation to BYOL |
|---|---|---|
| BGRL (Thakoor et al., ICLR 2022) | 2021 | Bootstrapped Graph Latents: learns graph representations by predicting alternative augmentations without negatives; one of the winning entries in the OGB Large-Scale Challenge at KDD Cup 2021[11] |
| BYOL-A (Niizumi et al., IJCNN 2021) | 2021 | Adapts BYOL to general-purpose audio representations learned from a single audio segment[12] |
| Barlow Twins (Zbontar et al., ICML 2021) | 2021 | Groups BYOL and SimSiam as methods that avoid collapse through architectural and update asymmetry; reports that its own method, unlike BYOL, is not robust to removing some augmentations[10] |
| DINO (Caron et al.) | 2021 | Says it takes inspiration from BYOL but uses a different similarity loss and the same architecture for student and teacher, interpreting self-supervised learning as Mean Teacher self-distillation without labels[9] |
| BYOL-Explore (Guo et al.) | 2022 | Curiosity-driven exploration from DeepMind that learns a world representation, dynamics and exploration policy with one latent prediction loss; reported superhuman performance on the ten hardest-exploration Atari games[13] |
Several of the BGRL and BYOL-Explore authors, including Corentin Tallec, Rémi Munos, Michal Valko, Mohammad Gheshlaghi Azar and Bilal Piot, were also BYOL authors.[1][11][13]
Code and checkpoints
DeepMind released a JAX implementation of BYOL's pre-training and linear evaluation pipeline in its deepmind-research repository.[3] The README states that it should reach between 74.0% and 74.5% top-1 on ImageNet after about 8 hours on 512 Cloud TPU v3, and provides pre-trained checkpoints for ResNet-50 (1x), expected to evaluate to about 74.4%, and ResNet-200 (2x), about 79.6%, along with checkpoints for the batch-size and augmentation ablations.[3] It warns that the paper's experiments were run on Google's internal infrastructure and does not recommend the full experiment on public Cloud TPUs because data loading was a bottleneck; a small-scale setup on the Imagenette dataset with a single V100 GPU was offered instead.[3] The code is licensed under the Apache License 2.0, while the checkpoint weights are licensed CC BY-NC 4.0 for non-commercial use.[3]
Limitations
- Vision-specific augmentations. The authors wrote that BYOL depends on augmentation sets designed for vision, and that extending it to audio, video or text would require comparably suitable augmentations, which may take significant effort and expertise.[1]
- Gap to strong supervised baselines. The ResNet-50 result remained below the 78.9% supervised baseline the paper cited.[1]
- No guarantee against collapse. The paper's account of why BYOL avoids collapse was a hypothesis supported by experiments, and the dynamics "still admit undesirable equilibria."[1]
- Dataset bias. The broader-impact statement noted that representations learned from vision datasets could replicate those datasets' biases.[1]
Continue reading
- SimCLRSimCLR (Simple Framework for Contrastive Learning of Visual Representations) is a self-supervised learning method for computer vision in which a network is trained to recognise that two differently augmented…Mentioned 20 times above
- Target NetworkA target network is a separate, slowly updated copy of a neural network used in deep reinforcement learning to compute stable learning targetsMentioned 12 times above
- MoCo (Momentum Contrast)MoCo (Momentum Contrast) is a family of self-supervised learning methods for learning visual representations from unlabeled images, developed at Facebook AI Research (FAIR, now Meta FAIR).Mentioned 8 times above
- Self-Supervised LearningSelf-supervised learning (SSL) is a machine learning approach in which a model learns representations from unlabeled data by generating its own supervisory signal from the structure of the data itselfMentioned 3 times above
- Contrastive LearningContrastive learning is a family of machine learning methods that learn representations by pulling similar (positive) pairs of data points closer together in an embedding space while pushing dissimilar…Mentioned twice above
- DINO (computer vision)DINO (self-DIstillation with NO labels) is a family of self-supervised learning methods for computer vision from Meta AI that trains Vision Transformers (ViTs) on unlabeled images and produces general-purpose…Mentioned twice above
More from See also
More to read
Topics:Computer VisionDeep LearningGoogle DeepMindMachine Learning
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51 ^52 ^53 ^54 ^55 ^56 ^57 ^58 ^59 ^60 ^61 ^62Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko. "Bootstrap your own latent: A new approach to self-supervised Learning". arXiv:2006.07733, 13 June 2020, revised 10 September 2020. arxiv.org/...2006.07733
- ^Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, et al. "Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning". Advances in Neural Information Processing Systems 33 (NeurIPS 2020). proceedings.neurips.cc/...2b17b8192b2958e-Abstract
- ^1 ^2 ^3 ^4Google DeepMind. "Bootstrap Your Own Latent" (README). GitHub, google-deepmind/deepmind-research, byol directory. github.com/...byol
- ^1 ^2 ^3Josh Albrecht, Abe Fetterman. "Understanding self-supervised and contrastive learning with 'Bootstrap Your Own Latent' (BYOL)". Imbue, 24 August 2020. imbue.com/...-self-supervised-contrastive-learning
- ^1 ^2 ^3 ^4 ^5 ^6Pierre H. Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, Michal Valko. "BYOL works even without batch statistics". arXiv:2010.10241, 20 October 2020. arxiv.org/...2010.10241
- ^1 ^2 ^3Xinlei Chen, Kaiming He. "Exploring Simple Siamese Representation Learning". arXiv:2011.10566, 20 November 2020. arxiv.org/...2011.10566
- ^Xinlei Chen, Kaiming He. "Exploring Simple Siamese Representation Learning". 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753, June 2021. DOI 10.1109/CVPR46437.2021.01549. doi.org/...CVPR46437.2021.01549
- ^1 ^2 ^3Yuandong Tian, Xinlei Chen, Surya Ganguli. "Understanding self-supervised learning dynamics without contrastive pairs". Proceedings of the 38th International Conference on Machine Learning, PMLR 139, pp. 10268-10278, 2021. proceedings.mlr.press/...tian21a
- ^1 ^2Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin. "Emerging Properties in Self-Supervised Vision Transformers". arXiv:2104.14294, 29 April 2021. arxiv.org/...2104.14294
- ^Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, Stéphane Deny. "Barlow Twins: Self-Supervised Learning via Redundancy Reduction". arXiv:2103.03230 (ICML 2021), 4 March 2021. arxiv.org/...2103.03230
- ^1 ^2 ^3Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L. Dyer, Rémi Munos, Petar Veličković, Michal Valko. "Large-Scale Representation Learning on Graphs via Bootstrapping". arXiv:2102.06514 (ICLR 2022), 12 February 2021. arxiv.org/...2102.06514
- ^1 ^2Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Kunio Kashino. "BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation". arXiv:2103.06695 (IJCNN 2021), 11 March 2021. arxiv.org/...2103.06695
- ^1 ^2 ^3Zhaohan Daniel Guo, Shantanu Thakoor, Miruna Pîslar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastien Grill, Yunhao Tang, Michal Valko, Rémi Munos, Mohammad Gheshlaghi Azar, Bilal Piot. "BYOL-Explore: Exploration by Bootstrapped Prediction". arXiv:2206.08332, 16 June 2022. arxiv.org/...2206.08332
- ^Xinlei Chen, Saining Xie, Kaiming He. "An Empirical Study of Training Self-Supervised Vision Transformers". arXiv:2104.02057 (ICCV 2021), 5 April 2021. arxiv.org/...2104.02057
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
2 revisions · v2 · 3,429 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check before publication on November 18, 2026: 118 claims checked against 15 cited and independent sources; 0 material and 4 minor corrections applied before publication.
Cite this page: AI Wiki. "BYOL (Bootstrap Your Own Latent)." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/byol

