Adafactor

Adafactor is an adaptive optimizer designed to reduce the memory used by second-moment estimates during neural-network training. Noam Shazeer and Mitchell Stern introduced it at ICML 2018 in Adafactor: Adaptive Learning Rates with Sublinear Memory Cost.^[1] For an $n \times m$ parameter matrix, the method replaces the full $n \times m$ second-moment accumulator used by Adam with running row and column statistics containing $n+m$ values. Vectors still use an unfactored accumulator.^[1]

The paper combines that factorization with update clipping, a second-moment decay rate that changes with the training step, and a step size scaled to the root-mean-square (RMS) magnitude of each parameter tensor. Its proposed low-memory configuration omits momentum, although a full first-moment accumulator can be added.^[1] These choices are separable. An implementation described as "Adafactor" may use the factorization, parameter scaling, update clipping, momentum, or only some of them.

Adafactor was evaluated originally on a Transformer machine translation task. Later, T5 used it for pretraining and fine-tuning, while PaLM used an unfactored version with momentum.^[1][3][4] Those examples establish practical use at large scale, but they do not show that Adafactor is universally more accurate or faster than Adam-family optimizers.

Design and memory model

Relationship to Adam

AdaGrad, RMSProp, and Adam are adaptive machine learning methods that normalize a gradient using statistics derived from current or past squared gradients. Adam keeps a full first-moment estimate and a full second-moment estimate for every parameter.^[2] AdamW changes how weight decay is applied but retains Adam's two moment estimates.^[10]

For a matrix with $nm$ parameters, Adam therefore stores $2nm$ moment values. The optimizer state alone is twice the size of the parameter array when all three use the same numeric type; parameters plus the two moment arrays occupy three such arrays. At 32 bits per value, the two Adam moment buffers for one billion parameters contain about 8 GB of data in decimal units. The second-moment buffer by itself contains about 4 GB. These figures exclude gradients, activations, master-weight copies, allocator overhead, and any distributed-training state.

Adafactor targets the second of those two moment arrays. Its proposed configuration also sets the first-moment decay to zero, which removes the first-moment array entirely.^[1] If momentum is restored in the Adam form, its accumulator is full size. The state for an $n \times m$ matrix then becomes $nm+n+m$ values, not $2(n+m)$. If factorization is also disabled, as in the PaLM configuration, both moment arrays are full size and the moment-state count is again $2nm$.^[1][4]

Factored second-moment estimate

Let $G_t$ be the gradient of a matrix parameter at training step $t$. The original algorithm maintains a column of row sums $R_t$ and a row of column sums $C_t$ for an exponential moving average of $G_t^2$. It approximates the full nonnegative second-moment matrix as

$$\hat{V}_t = \frac{R_t C_t}{\sum_i R_{t,i}}.$$

The denominator can equivalently be written as the sum of the column statistics, so the construction does not favor rows over columns.^[1] The paper derives this outer product as the rank-1 nonnegative approximation that minimizes generalized Kullback-Leibler divergence, also called I-divergence, from the full second-moment matrix. Row and column sums are linear, which allows the optimizer to update the factors directly under exponential averaging without first storing the full matrix.^[1]

The unscaled adaptive update is

$$U_t = \frac{G_t}{\sqrt{\hat{V}_t}}.$$

For a vector parameter, the paper instead keeps a full elementwise second-moment estimate and applies the same normalization.^[1] Frameworks have different policies for higher-rank tensors and small matrices, so the actual state size depends on tensor shapes and the implementation's factoring threshold.^[7][8]

State-size example

For a $4096 \times 16384$ matrix:

Scroll sideways for more →

For this tensor, factoring the second moment reduces that accumulator's persistent state by about 3,277 times. Enabling momentum restores the entire cost of Adam's full first-moment array, so the remaining reduction comes from the factored second moment. There is no factorization saving for vectors, and the saving is smaller for narrow matrices. It also applies only to optimizer state, not to the model parameters, forward activations, or gradients.^[1]

Update rule

Update clipping

A slowly changing second-moment estimate can become stale when gradient magnitudes change. The original paper observed normalized updates with RMS values far above their intended scale and proposed clipping the update after adaptive normalization:

$$\hat{U}_t = \frac{U_t}{\max(1,\operatorname{RMS}(U_t)/d)}.$$

The proposed threshold is $d=1$.^[1] This differs from gradient clipping. Gradient clipping bounds a norm before the optimizer's coordinate-wise scaling, while update clipping acts on the optimizer's normalized update. A separately clipped gradient can still produce a large adaptive update.^[1]

Increasing second-moment decay

Adafactor's paper proposes

$$\hat{\beta}_{2,t}=1-t^{-c},$$

where $0<c\leq1$. The coefficient starts at zero and approaches one, so the accumulator changes quickly early in training and more slowly later. The paper recommends $c=0.8$ and shows that this schedule does not need a separate zero-initialization bias correction.^[1]

API conventions obscure the sign of this parameter. PyTorch, Keras, and Hugging Face expose a value of -0.8 and compute a power of the step from it, while Optax exposes decay_rate=0.8.^[5][6][7][8] Copying a numeric value between frameworks without checking the definition can therefore change the schedule.

Parameter-relative step size

The paper defines the effective learning rate for a tensor as

$$\alpha_t=\max(\epsilon_2,\operatorname{RMS}(X_{t-1}))\rho_t,$$

with proposed relative schedule

$$\rho_t=\min(0.01,t^{-1/2}).$$

The floor $\epsilon_2=10^{-3}$ lets zero-initialized or very small parameters move. The recommended squared-gradient regularizer is $\epsilon_1=10^{-30}$, and the recommended update-clipping threshold is one.^[1] Parameter scaling makes the absolute update depend on a tensor's current magnitude. It is not the same as choosing a single global learning rate.

The paper also evaluated a warmup form of the relative schedule. Current libraries expose different combinations of an external learning rate, internal relative scheduling, parameter scaling, and warmup.^[5][6][7][8] A configuration is reproducible only if those choices and the specific implementation are recorded.

Matrix update sequence

For the paper's no-momentum matrix configuration, one training step can be summarized as follows:^[1]

1. Compute the gradient $G_t$.
2. Update the row and column moving averages of $G_t^2+\epsilon_1$.
3. Form the factored estimate $\hat{V}_t$.
4. Normalize the gradient to obtain $U_t$.
5. Clip the RMS of $U_t$ at threshold $d$.
6. Multiply by the parameter-relative step size $\alpha_t$.
7. Subtract the result from the parameter.

Weight decay is not part of the paper's proposed algorithm. Libraries that offer it make their own choices about where and how it enters the update.^{[1][5][6][7][8]}

Evidence in the original paper

The 2018 study trained the Transformer from Attention Is All You Need on the WMT 2014 English-to-German task. Its main experiments ran for 100,000 steps and compared full and factored second moments, with and without momentum, update clipping, warmup, variable decay, and relative step sizes.^[1]

Factoring the second moment produced similar results to the full accumulator in the tested settings. Removing momentum without other changes was unstable when warmup was absent. Update clipping at $d=1$ and a changing second-moment decay reduced that instability. A factored, no-momentum configuration with the proposed decay schedule and update clipping reached results comparable to the paper's Adam baseline while using $O(n+m)$ state for each matrix and full state for vectors.^[1]

The scope of that evidence matters. It came from one Transformer machine-translation setup, not a broad optimizer benchmark across model families and tasks. The paper did not establish that factorization always preserves convergence, that Adafactor needs no tuning, or that it improves final quality. Its supported conclusion is narrower: the tested low-memory configurations matched the study's Adam regime closely enough to make the state reduction useful.^[1]

Implementations

Adafactor is available in current PyTorch, TensorFlow and Keras, Hugging Face Transformers, and JAX through Optax.^{[5][6][7][8][13]} Their APIs are not interchangeable.

Scroll sideways for more →

Hugging Face documents support for FP16 and bfloat16 parameter values but also says that support has not been thoroughly tested.^[6] Optax lets users choose the momentum-buffer dtype when momentum is enabled.^[7] Keras offers loss scaling through its optimizer infrastructure.^[8] These features concern numerical representation and framework integration; they do not change the mathematical reason that row and column state uses less memory than a full second-moment matrix.

Configuration consequences

Three settings account for many apparent contradictions between Adafactor recipes:

• Factoring: Turning it off removes the defining second-moment memory saving.
• Momentum: Turning it on adds a full first-moment accumulator in the usual formulation.^[1]
• Step control: An internal relative schedule and an external schedule are different regimes. Hugging Face documents disabling relative steps and parameter scaling when its Adafactor receives a manual external schedule.^[6]

The optimizer name alone is therefore insufficient for reproducing a run. At minimum, a report should state the library and version, factoring policy, first-moment setting, second-moment decay convention, clipping threshold, parameter-scaling choice, learning-rate schedule, epsilon values, weight-decay rule, and numeric types.

Use in large-model training

T5

The T5 study used Adafactor throughout its text-to-text experiments.^[3] Its baseline pretraining ran with an inverse-square-root schedule that held the rate at 0.01 for the first 10,000 steps and then decayed it. Fine-tuning used a constant rate of 0.001.^[3] This is more specific than saying that T5 used Adafactor with no external schedule. The published T5 procedure included explicit pretraining and fine-tuning schedules.

T5 is a documented large-scale use of Adafactor in a transfer learning workflow. It does not establish one universal T5 fine-tuning recipe. Hugging Face documents both an external-rate configuration and a relative-step configuration, labels some recommendations as community-derived, and warns against combining its Adafactor settings with additional optimizer operations such as separate gradient clipping.^[6]

PaLM

PaLM's 540-billion-parameter training run used Adafactor without factorization.^[4] The PaLM paper calls this effectively Adam with parameter scaling. It enabled momentum with $\beta_1=0.9$, used the step-dependent second-moment coefficient $1-k^{-0.8}$, applied global-norm gradient clipping at one, and used a learning rate of $10^{-2}$ for the first 10,000 steps followed by inverse-square-root decay.^[4] The run used 6,144 Cloud TPU v4 accelerators through Pathways.^[4]

PaLM is consequently evidence for Adafactor's parameter-scaling and decay-schedule ideas, but not for factored optimizer-state savings. It also shows why adoption lists need configuration detail: two projects can both report "Adafactor" while retaining very different amounts of optimizer state.

Limitations and interpretation

• The saving depends on tensor shape. Matrices receive the largest benefit. Vectors retain a full second moment, and framework thresholds may leave small matrices unfactored.^[1][7]
• Momentum changes the memory calculation. A full first moment costs one value per parameter. Enabling it does not produce a second pair of row and column factors.^[1]
• Factorization is an approximation. It preserves row and column totals under the paper's rank-1 construction, not every entry of the full second-moment matrix.^[1]
• Implementation differences affect results. Learning-rate semantics, epsilon placement, factoring thresholds, weight decay, and higher-rank handling vary across maintained libraries.^[5][6][7][8]
• The original evidence is narrow. Comparable WMT results do not prove equal convergence on small models, short fine-tuning runs, vision tasks, or every large language model.^[1]
• Memory reduction does not imply proportional speedup. The optimizer still computes gradients and normalized elementwise updates. It reduces persistent optimizer state, while total training time also depends on model computation, communication, memory bandwidth, and framework kernels.

Later work has treated the approximation error as a real optimization tradeoff. The CAME paper used Adafactor as a memory-efficient baseline, reported a performance penalty in its tested language-model settings, and added confidence-guided state intended to improve stability and convergence.^[9] That result does not invalidate the original WMT finding; it shows that the tradeoff depends on the model, task, and configuration.

Other memory-saving approaches act at different points. GaLore projects gradients into a lower-rank subspace while still updating full-rank parameters.^[11] LoRA and QLoRA reduce the trainable parameter set during adaptation.^[14][15] The ZeRO method implemented in DeepSpeed partitions optimizer state across data-parallel processes.^[12] These methods can complement or replace factored second moments, but their memory accounting and optimization behavior are different.

See also

• mT5
• Switch Transformer
• Mixture of Experts
• Google Brain
• Stochastic Gradient Descent (SGD)
• Lion optimizer
• Sophia optimizer
• Muon optimizer
• Shampoo optimizer
• Schedule-Free optimizer

References

1. ^Noam Shazeer and Mitchell Stern, "Adafactor: Adaptive Learning Rates with Sublinear Memory Cost," Proceedings of the 35th International Conference on Machine Learning, PMLR 80, 2018. proceedings.mlr.press/...shazeer18a
2. ^Diederik P. Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization," ICLR 2015. arxiv.org/...1412.6980
3. ^Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," Journal of Machine Learning Research 21(140), 2020. jmlr.org/...20-074
4. ^Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, et al., "PaLM: Scaling Language Modeling with Pathways," Journal of Machine Learning Research 24(240), 2023. jmlr.org/...22-1144
5. ^PyTorch, "torch.optim.Adafactor," PyTorch documentation, accessed July 28, 2026. docs.pytorch.org/...torch.optim.Adafactor
6. ^Hugging Face, "Optimization: Adafactor," Transformers documentation, accessed July 28, 2026. huggingface.co/...optimizer_schedules
7. ^Optax, "optax.adafactor," Optax documentation, accessed July 28, 2026. optax.readthedocs.io/...optax.adafactor
8. ^Keras, "Adafactor," Keras 3 API documentation, accessed July 28, 2026. keras.io/...adafactor
9. ^Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, and Yang You, "CAME: Confidence-guided Adaptive Memory Efficient Optimization," Proceedings of ACL 2023, pages 4442-4453. aclanthology.org/2023.acl-long.243
10. ^Ilya Loshchilov and Frank Hutter, "Decoupled Weight Decay Regularization," ICLR 2019. openreview.net/forum
11. ^Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian, "GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection," Proceedings of the 41st International Conference on Machine Learning, PMLR 235, 2024. proceedings.mlr.press/...zhao24s
12. ^Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He, "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models," SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 2020. arxiv.org/...1910.02054
13. ^TensorFlow, "tf.keras.optimizers.Adafactor," TensorFlow API documentation, accessed July 28, 2026. tensorflow.org/...Adafactor
14. ^Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, "LoRA: Low-Rank Adaptation of Large Language Models," ICLR 2022. arxiv.org/...2106.09685
15. ^Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs," Advances in Neural Information Processing Systems 36, 2023. proceedings.neurips.cc/...049b-Abstract-Conference