Dust
Dust is a zeroth-order training method proposed by Q Labs in October 2026 for language transformers without backpropagation.[1]
Training approach
Independent activation perturbations at each token form a virtual population. Reward-weighted noise supplies estimated updates. Selected attention internals receive decayed future-token credit. The residual-mixing scalars instead use weight-space evolution strategies.[1][2]
Node perturbation predates Dust. Werfel, Xie and Seung's 2003 analysis distinguishes perturbing weights from perturbing unit outputs. In their linear-perceptron setting, the latter uses fewer noise dimensions. An error change weights the output noise, and its outer product with the input supplies a weight update. Their learning-rate comparisons apply to that analyzed model, not directly to transformer pretraining.[3]
Experiments
The base model has eight layers and width 512. FineWeb training spans 100k-20M tokens, one epoch, three seeds and populations of 64-16k draws.[1]
At nominal 16k population, validation-selected test losses were:[1]
| Training tokens | Dust | Backpropagation |
|---|---|---|
| 10M | 5.049 | 4.989 |
| 20M | 4.802 | 4.633 |
Claims and limitations
The report's - comparison extrapolates EGGROLL-Transformer populations, not a measured backpropagation speedup. The 1B-token gradient diagnostic uses backprop-trained checkpoints, not Dust-trained models. The authors say practical replacement requires orders-of-magnitude efficiency improvements.[1]
Released implementation
The MIT-licensed minimal implementation omits execution optimizations used in the experiments.[2]
Related approaches
These methods address different problems or use different information when estimating an update. Their results are not interchangeable evidence of a universal training speedup.
| Approach | Relevant distinction |
|---|---|
| EGGROLL | Uses low-rank perturbations of model parameters, rather than perturbing activations. Its original experiments include integer recurrent-language-model pretraining and language-model reasoning post-training.[4] |
| MeZO | Uses paired forward evaluations with parameter perturbations and reconstructs noise from a random seed to reduce memory. Its language-model experiments concern fine-tuning pretrained models, not learning representations from scratch.[5] |
| Forward gradient with local losses | Ren and colleagues combine activation perturbations with local losses. They use forward-mode automatic differentiation to obtain directional derivatives, so avoiding a backward pass does not make their method zeroth-order.[6] |
Continue reading
- BackpropagationBackpropagation is an algorithm for computing derivatives of a scalar objective with respect to variables in a layered or otherwise composed computation.Mentioned 3 times above
- TransformersA Transformer is a deep learning architecture, introduced by eight Google researchers in the 2017 paper "Attention Is All You Need", that uses attention as the sole mechanism for modeling relationships between…Linked in the introduction
- Fine-tuningFine-tuning is the process of adapting a pretrained machine-learning model by continuing to optimize some or all of its parameters on data selected for a target task, domain, behavior, or population.Also in Training & Optimization
- LIMA (Less Is More for Alignment)LIMA, short for "Less Is More for Alignment," is a 2023 research paper by Chunting Zhou and colleagues at Meta AI, Carnegie Mellon University, the University of Southern California, and Tel Aviv University…Also in Training & Optimization
- Intrinsic Distribution MatchingIntrinsic Distribution Matching (IDM) is a method for training image generators that use one or two generator evaluations.Also in Training & Optimization
- Weight-Space ProgrammingWeight-Space Programming is an approach in which a general-purpose large language model reads, interprets, edits or constructs another neural network's numerical weights in response to a natural-language goalAlso in Training & Optimization
More to read
References
- ^1 ^2 ^3 ^4 ^5Dahal, S., Mandal, B., Gülbahar, S., and Vegesna, A. (2026). Dust: Pretraining Transformers Without Backpropagation. Q Labs Research, October. Includes linked appendix.
- ^1 ^2Q Labs. Dust implementation. README, dust.py and model.py, inspected October 12, 2026.
- ^Werfel, J., Xie, X., and Seung, H. S. (2003). Learning Curves for Stochastic Gradient Descent in Linear Feedforward Networks. *Advances in Neural Information Processing Systems*, 16.
- ^Sarkar, B., et al. (2026 revision). Evolution Strategies at the Hyperscale. Version 2.
- ^Malladi, S., et al. (2023). Fine-Tuning Language Models with Just Forward Passes. *Advances in Neural Information Processing Systems*, 36.
- ^Ren, M., Kornblith, S., Liao, R., and Hinton, G. (2023). Scaling Forward Gradient With Local Losses. *International Conference on Learning Representations*.
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
3 revisions · v3 · 475 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent full-article review against the cited primary and academic sources, October 12, 2026; supported math rendering checked after delimiter-only formatting repair.
Cite this page: AI Wiki. "Dust." aiwiki.ai, updated 11 Oct 2026, fact-checked 11 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/dust

