# Dust

> Source: https://aiwiki.ai/wiki/dust
> Updated: 2026-10-11
> Fact-checked: 2026-10-11
> Categories: AI Research, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Dust." aiwiki.ai, 11 Oct 2026. https://aiwiki.ai/wiki/dust
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Dust** is a zeroth-order training method proposed by Q Labs in October 2026 for language [transformers](https://aiwiki.ai/wiki/transformers) without [backpropagation](https://aiwiki.ai/wiki/backpropagation).[1]

## Training approach

Independent activation perturbations at each token form a virtual population. Reward-weighted noise supplies estimated updates. Selected attention internals receive decayed future-token credit. The $$2L$$ residual-mixing scalars instead use weight-space evolution strategies.[1][2]

Node perturbation predates Dust. Werfel, Xie and Seung's 2003 analysis distinguishes perturbing weights from perturbing unit outputs. In their linear-perceptron setting, the latter uses fewer noise dimensions. An error change weights the output noise, and its outer product with the input supplies a weight update. Their learning-rate comparisons apply to that analyzed model, not directly to transformer pretraining.[3]

## Experiments

The base model has eight layers and width 512. [FineWeb](https://aiwiki.ai/wiki/fineweb) training spans 100k-20M tokens, one epoch, three seeds and populations of 64-16k draws.[1]

At nominal 16k population, validation-selected test losses were:[1]

| Training tokens | Dust | Backpropagation |
| --- | --- | --- |
| 10M | 5.049 | 4.989 |
| 20M | 4.802 | 4.633 |

## Claims and limitations

The report's $$10^3$$-$$10^4$$ comparison extrapolates EGGROLL-Transformer populations, not a measured backpropagation speedup. The 1B-token gradient diagnostic uses backprop-trained checkpoints, not Dust-trained models. The authors say practical replacement requires orders-of-magnitude efficiency improvements.[1]

## Released implementation

The MIT-licensed [minimal implementation](https://github.com/qlabs-eng/dust/tree/b20f7c03eac630b6441ba6f254128c1761dc45ad) omits execution optimizations used in the experiments.[2]

## Related approaches

These methods address different problems or use different information when estimating an update. Their results are not interchangeable evidence of a universal training speedup.

| Approach | Relevant distinction |
| --- | --- |
| EGGROLL | Uses low-rank perturbations of model parameters, rather than perturbing activations. Its original experiments include integer recurrent-language-model pretraining and language-model reasoning post-training.[4] |
| MeZO | Uses paired forward evaluations with parameter perturbations and reconstructs noise from a random seed to reduce memory. Its language-model experiments concern [fine-tuning](https://aiwiki.ai/wiki/fine_tuning) pretrained models, not learning representations from scratch.[5] |
| Forward gradient with local losses | Ren and colleagues combine activation perturbations with local losses. They use forward-mode [automatic differentiation](https://aiwiki.ai/wiki/automatic_differentiation) to obtain directional derivatives, so avoiding a backward pass does not make their method zeroth-order.[6] |

## References

1. Dahal, S., Mandal, B., Gülbahar, S., and Vegesna, A. (2026). [Dust: Pretraining Transformers Without Backpropagation](https://qlabs.sh/research/dust). Q Labs Research, October. Includes linked appendix.

2. Q Labs. [Dust implementation](https://github.com/qlabs-eng/dust/tree/b20f7c03eac630b6441ba6f254128c1761dc45ad). README, dust.py and model.py, inspected October 12, 2026.

3. Werfel, J., Xie, X., and Seung, H. S. (2003). [Learning Curves for Stochastic Gradient Descent in Linear Feedforward Networks](https://proceedings.neurips.cc/paper/2003/file/f8b932c70d0b2e6bf071729a4fa68dfc-Paper.pdf). *Advances in Neural Information Processing Systems*, 16.

4. Sarkar, B., et al. (2026 revision). [Evolution Strategies at the Hyperscale](https://arxiv.org/abs/2511.16652v2). Version 2.

5. Malladi, S., et al. (2023). [Fine-Tuning Language Models with Just Forward Passes](https://arxiv.org/abs/2305.17333). *Advances in Neural Information Processing Systems*, 36.

6. Ren, M., Kornblith, S., Liao, R., and Hinton, G. (2023). [Scaling Forward Gradient With Local Losses](https://arxiv.org/abs/2210.03310). *International Conference on Learning Representations*.