Dust

Dust is a zeroth-order training method proposed by Q Labs in October 2026 for language transformers without backpropagation.^[1]

Training approach

Independent activation perturbations at each token form a virtual population. Reward-weighted noise supplies estimated updates. Selected attention internals receive decayed future-token credit. The $2L$ residual-mixing scalars instead use weight-space evolution strategies.^[1][2]

Node perturbation predates Dust. Werfel, Xie and Seung's 2003 analysis distinguishes perturbing weights from perturbing unit outputs. In their linear-perceptron setting, the latter uses fewer noise dimensions. An error change weights the output noise, and its outer product with the input supplies a weight update. Their learning-rate comparisons apply to that analyzed model, not directly to transformer pretraining.^[3]

Experiments

The base model has eight layers and width 512. FineWeb training spans 100k-20M tokens, one epoch, three seeds and populations of 64-16k draws.^[1]

At nominal 16k population, validation-selected test losses were:^[1]

Claims and limitations

The report's $10^3$-$10^4$ comparison extrapolates EGGROLL-Transformer populations, not a measured backpropagation speedup. The 1B-token gradient diagnostic uses backprop-trained checkpoints, not Dust-trained models. The authors say practical replacement requires orders-of-magnitude efficiency improvements.^[1]

Released implementation

The MIT-licensed minimal implementation omits execution optimizations used in the experiments.^[2]

Related approaches

These methods address different problems or use different information when estimating an update. Their results are not interchangeable evidence of a universal training speedup.

References

1. ^1 ^2 ^3 ^4 ^5Dahal, S., Mandal, B., Gülbahar, S., and Vegesna, A. (2026). Dust: Pretraining Transformers Without Backpropagation. Q Labs Research, October. Includes linked appendix.
2. ^1 ^2Q Labs. Dust implementation. README, dust.py and model.py, inspected October 12, 2026.
3. ^Werfel, J., Xie, X., and Seung, H. S. (2003). Learning Curves for Stochastic Gradient Descent in Linear Feedforward Networks. *Advances in Neural Information Processing Systems*, 16.
4. ^Sarkar, B., et al. (2026 revision). Evolution Strategies at the Hyperscale. Version 2.
5. ^Malladi, S., et al. (2023). Fine-Tuning Language Models with Just Forward Passes. *Advances in Neural Information Processing Systems*, 36.
6. ^Ren, M., Kornblith, S., Liao, R., and Hinton, G. (2023). Scaling Forward Gradient With Local Losses. *International Conference on Learning Representations*.