Learning Rate

The learning rate is a hyperparameter that scales an update made by an iterative optimization algorithm. In machine learning, it is commonly written as η or α. For ordinary gradient descent, the update at iteration t is

$$\theta_{t+1} = \theta_t - \eta_t \nabla F(\theta_t).$$

where θₜ is the parameter vector, F is the objective or loss function, and ηₜ is the learning rate. The term step size is more common in numerical optimization; learning rate is conventional in machine learning. A constant learning rate uses the same η at every iteration, while a learning-rate schedule makes ηₜ depend on the iteration, epoch, validation history, or another signal ^[1].

The numerical value is not meaningful in isolation. Its effect depends on the objective's scale and curvature, the model's parameterization, the optimizer, momentum or adaptive preconditioning, batch construction, regularization, and the schedule. Consequently, there is no learning rate that is generally correct for "Adam," "transformers," or models of a given parameter count. Bengio's practical review called the choice of learning rate crucial, but its recommendations were explicitly empirical rather than universal constants ^[2].

Update size and effective learning rate

A useful generic first-order update is

$$\theta_{t+1} = \theta_t - \eta_t d_t.$$

where dₜ is the algorithm's update direction. For full-batch gradient descent, dₜ is the exact gradient. For stochastic gradient descent (SGD), dₜ is estimated from a sample or mini-batch. Momentum replaces the current gradient with an accumulated direction. Adaptive methods rescale coordinates using gradient history. Gradient clipping, normalization layers, and weight decay can further change the resulting parameter displacement ^[1].

This distinction motivates the phrase effective learning rate. It may refer to the magnitude of the actual update relative to the parameter, or to a coordinate-wise rate after preconditioning. There is no single standardized definition across the literature. For example, Adam has a scalar base learning rate, but divides its first-moment estimate by a coordinate-wise function of its second-moment estimate ^[4]. Comparing only the scalar learning-rate fields of two different optimizers can therefore be misleading.

Dependence on loss and parameter scale

The same numerical η can describe different optimization problems. Multiplying the entire objective by a positive constant c multiplies its gradient by c, so gradient descent at rate η on cF makes the same update as rate cη on F. A loss implementation that sums examples therefore has a different gradient scale from one that averages the same examples. Changing units, target normalization, or the relative weights of component losses can have a similar effect.

Parameterization matters as well. Two networks can initially compute the same function while assigning different scales to their parameters and gradients. Normalization and scale symmetries can make the parameter norm a poor proxy for functional change. This is one reason that learning rates do not transfer automatically between codebases or between a standard parameterization and a scale-aware one such as μP ^[18].

Momentum example

One common SGD-with-momentum convention is

$$\begin{aligned} v_{t+1} &= \beta v_t + g_t, \\ \theta_{t+1} &= \theta_t - \eta_t v_{t+1}. \end{aligned}$$

where gₜ is a stochastic gradient and β is the momentum coefficient. With a constant η, an algebraically equivalent convention places η inside the velocity recurrence and subtracts the velocity directly, so its stored velocity is scaled differently. If η changes over time, moving it into the recurrence also changes how earlier gradients are weighted unless the stored state is rescaled. Reproducible reports should identify the optimizer implementation and its convention rather than state only "SGD with learning rate η."

What theory says

Learning-rate guarantees are conditional on mathematical assumptions. If F has an L-Lipschitz gradient, the descent lemma gives

$$F\!\left(\theta - \eta \nabla F(\theta)\right) \le F(\theta) - \eta\left(1 - \frac{L\eta}{2}\right) \left\lVert\nabla F(\theta)\right\rVert_2^2.$$

Thus, for exact gradient descent, a positive η below 2/L gives a one-step decrease whenever the gradient is nonzero. Common convergence statements use η ≤ 1/L for a simpler margin. For a convex quadratic objective, the upper stable step-size limit is governed by the largest eigenvalue of its Hessian. These results do not provide a directly measurable universal bound for a changing, nonconvex neural-network objective ^[1].

Stochastic gradients add sampling noise, so even a convergent method need not reduce the sampled loss at every step. Classical stochastic-approximation results often use diminishing rates satisfying

$$\sum_{t=1}^{\infty}\eta_t = \infty \quad\text{and}\quad \sum_{t=1}^{\infty}\eta_t^2 < \infty,$$

together with assumptions on the objective and gradient noise. Finite-budget machine-learning training frequently uses other schedules because its goal is good validation performance after a specified amount of computation, not only asymptotic convergence ^[1].

The relevant guarantee also depends on the target. Convex analyses may bound objective suboptimality, strongly convex analyses may establish a linear rate under additional conditions, and nonconvex analyses often bound a gradient norm rather than prove arrival at a global minimum. An empirical statement that a model reached a useful validation score is different from each of these mathematical conclusions.

Related step-selection concepts

Several mechanisms are sometimes grouped under "adaptive learning rate," although they solve different problems:

• A schedule changes a scalar rate according to a rule such as step count or validation history.
• A preconditioner changes the relative scale of coordinates or parameter groups. AdaGrad and Adam combine a scalar base rate with diagonal preconditioning ^[3][4].
• A line search tests candidate step lengths against an objective-based condition. Classical line searches often assume access to a sufficiently reliable objective and direction; stochastic variants require additional design and assumptions ^[22].
• A trust-region method constrains a model of the local step rather than directly prescribing one scalar multiplier.
• Hyperparameter optimization treats the learning rate or schedule as an outer-loop choice evaluated through training runs. Hypergradient methods instead differentiate through or across updates ^[21].

These distinctions matter when an algorithm is advertised as "adaptive" or "learning-rate-free." Such wording may mean that it removes one manually chosen schedule while retaining a base scale, a bound, a tolerance, or another step-control parameter.

Too large and too small

A learning rate that is too large for a particular configuration can cause sustained oscillation, exploding parameters, non-finite arithmetic, or divergence. A smaller rate usually makes individual updates smaller, but can spend the available training budget making little progress. These symptoms are diagnostic clues, not proofs: numerical overflow, corrupted data, a faulty loss, or an incompatible precision configuration can resemble an excessive learning rate.

It is also inaccurate to say that a small learning rate inherently becomes "stuck in local minima." Optimization behavior depends on the geometry, stochastic noise, momentum, and training horizon. Learning rate can affect which solution a method reaches and can act as an implicit bias, but the simple large-steps-versus-local-minima story is not a general theorem.

Edge of stability

Deep-network training can depart from the monotone picture suggested by a fixed quadratic approximation. Cohen and colleagues observed that, in their full-batch neural-network experiments, the largest Hessian eigenvalue often rose to approximately 2/η. Training loss then oscillated over short intervals while declining over longer intervals, a regime they called the edge of stability ^[14]. This was an empirical result about full-batch gradient descent on the studied architectures and datasets; it does not establish that every mini-batch or adaptive-optimizer run should operate at the same threshold.

Development of modern practice

The learning-rate concept predates deep learning, but modern training combines ideas developed in different optimization settings.

Scroll sideways for more →

This chronology is not a ranking of optimizers or schedules. Later methods do not automatically supersede earlier ones, and results from one workload can depend on its tuning budget.

Learning-rate schedules

A schedule specifies the sequence η₀, η₁, … rather than a single number. It may require a planned training horizon T, respond to a metric, or adapt from optimization signals.

Scroll sideways for more →

Time base

A schedule's time coordinate must be specified. An epoch-based schedule advances after a pass through the training set. A step-based schedule advances after an optimizer update. A token-based schedule advances with the number of sequence tokens processed. These are not interchangeable when batch size, gradient accumulation, dataset sampling, or sequence length changes.

For example, doubling global batch size while keeping an epoch schedule fixed halves the number of optimizer updates per epoch. Keeping a step schedule fixed instead doubles the number of examples processed before the same schedule boundary. Comparisons that change batch size and silently retain only one of these time bases confound the learning rate with training exposure and update count ^[12].

Cosine annealing and restarts

For a cosine interval of length T, a common form is

$$\eta_t = \eta_{\min} + \frac{1}{2}\left(\eta_{\max} - \eta_{\min}\right) \left(1 + \cos\left(\frac{\pi t}{T}\right)\right).$$

SGDR proposed repeating such intervals and optionally increasing their lengths. The "warm restart" resets the learning rate, not the model parameters ^[9]. A monotone cosine decay without restarts is a related but distinct schedule. The existence of a dedicated cosine learning rate schedule article reflects that this schedule has implementation details beyond the learning-rate concept itself.

Cyclical and one-cycle policies

Smith's cyclical learning-rate work varied η between lower and upper bounds and proposed a short range test that increases the rate while monitoring loss ^[7]. Smith and Topin later reported "super-convergence" in selected image-classification experiments using a one-cycle policy with unusually large peak rates ^[8]. These papers provide empirical methods, not a guarantee that one cycle, a particular peak multiplier, or a specific cycle fraction is optimal for other architectures and data.

Horizon-dependent and schedule-free methods

Many decays use the planned stopping step T. Changing T alters the rate at every normalized position t/T, so extending a run can make a supposedly identical schedule a different experiment. Defazio and colleagues developed schedule-free SGD and AdamW variants that combine momentum and iterate averaging without a prespecified stopping time, and reported competitive results over their benchmark suite ^[16]. This approach removes one schedule dependency but does not make optimization hyperparameter-free.

Schaipp and colleagues studied a constant phase followed by a linear cooldown. They derived a related bound in a nonsmooth convex setting and reported experiments on 124-million- and 210-million-parameter Llama-type models, including schedule extension and transfer tests ^[17]. Those results are evidence for the studied scales and setup, not a general law for all large-language-model pretraining.

Warmup

Learning-rate warmup increases η from a smaller initial value to a target or peak value during an initial training interval. Warmup is a schedule component, not a separate optimizer.

Two influential examples had different contexts. The original Transformer used a learning rate proportional to

$$d_{\text{model}}^{-1/2} \min\left(\operatorname{step}^{-1/2}, \operatorname{step}\cdot \operatorname{warmup\_steps}^{-3/2}\right),$$

with 4,000 warmup steps in its reported experiments ^[10].

Goyal and colleagues used a linear-scaling heuristic for ResNet-50 on ImageNet: multiplying the mini-batch from 256 to 8,192 multiplied the learning rate from 0.1 to 3.2. Because applying the scaled rate immediately was unstable, they increased it gradually over the first five epochs ^[11].

These examples do not show that warmup is always necessary or that a fixed percentage is generally correct. In systematic experiments with SGD and Adam, Kalra and Barkeshli attributed much of warmup's benefit to allowing a larger target rate by moving training into better-conditioned regions. Their results also depended on initialization and parameterization, and their proposed changes shortened or, in some tested cases, eliminated warmup ^[15].

Warmup should be reported with its initial value, duration, curve, target rate, time unit, and the post-warmup schedule. "Ten warmup epochs" and "ten warmup steps" describe radically different procedures.

Adaptive learning rates

Adaptive optimizers such as AdaGrad and Adam form coordinate-wise scales from past gradients, but they retain a scalar base learning rate. They reduce some sensitivity to heterogeneous gradient magnitudes; they do not determine a universally optimal rate.

AdaGrad

AdaGrad accumulates squared gradients coordinate by coordinate and divides subsequent updates by the square root of that accumulation. Duchi, Hazan, and Singer derived regret and convergence properties and emphasized settings with sparse gradients, where frequently updated coordinates shrink differently from rare ones ^[3]. Because its accumulator does not forget, effective rates can keep decreasing; whether that is helpful depends on the problem.

Adam and AMSGrad

Adam combines exponential moving averages of the gradient and squared gradient with bias correction ^[4]. The original paper's convergence analysis was later shown to be insufficient: Reddi, Kale, and Kumar constructed a simple convex online problem on which Adam fails to converge and proposed AMSGrad, which retains a long-term maximum of the second-moment estimate ^[5]. This result does not imply that every Adam run diverges. It shows why optimizer name and default learning rate alone are not a convergence guarantee.

AdamW and weight decay

For ordinary SGD, L2 regularization and weight decay can be equivalent after accounting for learning-rate scaling. Loshchilov and Hutter showed that this equivalence does not hold for adaptive gradient methods and proposed AdamW, which decouples weight decay from the loss-gradient update. In their experiments, decoupling also made the best learning-rate and weight-decay choices more independent ^[6]. "Decoupled" does not mean the two hyperparameters never interact in every model; both remain part of the training configuration.

Relationship to batch size

Changing batch size changes the variance of the gradient estimate and the number of parameter updates made per pass through the data. Learning rate, batch size, momentum, and training duration therefore need to be considered together.

The linear scaling rule from the large-minibatch ResNet-50 work multiplies η by k when the batch size is multiplied by k ^[11]. Its approximation treats one large-batch update as similar to k smaller updates while the parameters and gradients change little. It was an empirical rule for a specified model, dataset, optimizer, and range, paired with warmup; it is not a theorem that applies unchanged to arbitrary batch sizes or adaptive optimizers.

A large empirical study by Shallue and colleagues trained 168,160 models across 35 workloads and found extremely large variation in the relationship between batch size and the number of steps to a target error. It also found that disagreements about large-batch generalization could largely be explained by different tuning procedures and compute budgets ^[12]. McCandlish and colleagues modeled diminishing parallel returns using a gradient-noise scale related to a problem's useful or "critical" batch size ^[13]. Together, these studies argue against a single universal linear or square-root scaling prescription.

Learning rate, training budget, and validation performance

Training loss, validation performance, wall-clock time, examples processed, and optimizer steps are different objectives. A rate that reduces training loss fastest in the first thousand updates may not give the best validation metric at a fixed epoch or the best result at a fixed compute budget. The ranking of rates can also change after decay. This is why a learning-rate study must define its stopping rule and selection metric ^[12].

Learning rate can influence implicit regularization, but broad claims such as "larger rates always find flatter minima" or "decay prevents overfitting" are not general results. Curvature measures depend on parameterization, and validation behavior is affected by the optimizer, batch noise, explicit regularization, data augmentation, and checkpoint selection. If the scientific question is generalization, candidates should be tuned to comparable training objectives and budgets rather than compared at a shared, untuned rate.

Selecting and tuning a learning rate

Learning-rate selection is an experiment-design problem. A defensible procedure defines the optimizer and schedule first, searches on training runs, selects using a validation set and objective, and keeps the test set out of tuning.

Common approaches include:

1. Log-spaced trials. Rates are multiplicative quantities, so a logarithmic search usually covers orders of magnitude more sensibly than an additive grid. Bergstra and Bengio found random search to be a reasonably efficient and reproducible baseline for hyperparameter optimization in their experiments, particularly when only some dimensions matter ^[20].
2. A range test. Increase η during a short run and observe where optimization improves and where it becomes unstable. Smith introduced this as a way to choose bounds for cyclical schedules ^[7]. It is a heuristic and requires an additional training run and compute; it does not prove that the selected value will be optimal for a full schedule.
3. Multi-fidelity tuning. Allocate smaller budgets to many candidates, then reserve the intended budget for the most promising configurations. Hyperband formalized this approach using resources such as iterations, data samples, or features together with adaptive allocation and early stopping ^[23]. Performance at low fidelity can misrank candidates, so finalists still require confirmation at the intended budget.
4. Online adaptation. Hypergradient descent differentiates an update with respect to the learning rate and adjusts it during training ^[21]. Stochastic Armijo line searches have also been studied under interpolation and other explicit assumptions ^[22]. Such methods trade a manually specified schedule for their own assumptions, state, or hyperparameters.

Validation comparisons should hold constant the data order policy, batch size, optimizer, model initialization policy, number of updates or examples processed, and stopping rule. Multiple seeds are useful when differences are comparable to run-to-run variation.

Reading training curves

Training curves can guide controlled tests, but a single curve rarely identifies one cause.

Scroll sideways for more →

A useful diagnostic changes one factor at a time over a broad enough multiplicative range to distinguish learning-rate sensitivity from implementation failure.

What to record

A reproducible learning-rate report should include:

• the initial, peak, minimum, and final rates where applicable;
• the complete schedule formula and whether time means updates, tokens, examples, or epochs;
• warmup and cooldown lengths and their curves;
• optimizer implementation, momentum or β values, numerical ε, and weight decay;
• global batch size, gradient accumulation, distributed-worker count, and any batch-size changes;
• clipping, freezing, layer-wise multipliers, and parameter groups;
• whether the schedule restarts or changes when training resumes from a checkpoint; and
• the validation metric, tuning budget, seed policy, and selected checkpoint.

Without this context, a quoted value such as 0.0003 is not a reproducible prescription.

Transfer across layers and model scale

Different parameter groups may use different learning rates. Howard and Ruder's ULMFiT method used discriminative fine-tuning, assigning lower rates to lower language-model layers, together with a slanted triangular schedule ^[19]. That result supports the method in its reported NLP transfer experiments; it does not establish that earlier layers must always use a fixed ratio in every fine-tuning task.

Model width and parameterization can also change the scale of useful updates. Yang and colleagues' maximal-update parameterization, μP, was designed so that several tuned hyperparameters remain stable as width changes. Their μTransfer experiments transferred near-optimal settings from smaller proxy networks to wider Transformer and ResNet targets ^[18]. The claim is conditional on implementing μP and on the axes of scale studied. It does not justify copying a learning rate between arbitrary architectures, depths, datasets, or standard parameterizations.

Interpretation

Learning rate is best understood as one component of an update rule rather than a stand-alone measure of how fast a model learns. Classical analysis connects it to curvature and gradient noise under stated assumptions. Modern practice adds schedules, warmup, adaptive preconditioners, and scale-dependent parameterizations. Empirical methods can narrow the search, but every numerical recommendation inherits the model, data, optimizer, implementation, and compute budget from which it was obtained.

References

1. ^Bottou, L., Curtis, F. E., and Nocedal, J. (2018). "Optimization Methods for Large-Scale Machine Learning." SIAM Review, 60(2), 223-311. Publisher record and author manuscript
2. ^Bengio, Y. (2012). "Practical Recommendations for Gradient-Based Training of Deep Architectures." Neural Networks: Tricks of the Trade, 2nd ed., 437-478. arXiv:1206.5533
3. ^Duchi, J., Hazan, E., and Singer, Y. (2011). "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization." Journal of Machine Learning Research, 12, 2121-2159. JMLR
4. ^Kingma, D. P., and Ba, J. (2015). "Adam: A Method for Stochastic Optimization." International Conference on Learning Representations. arXiv:1412.6980
5. ^Reddi, S. J., Kale, S., and Kumar, S. (2018). "On the Convergence of Adam and Beyond." International Conference on Learning Representations. OpenReview
6. ^Loshchilov, I., and Hutter, F. (2019). "Decoupled Weight Decay Regularization." International Conference on Learning Representations. OpenReview
7. ^Smith, L. N. (2017). "Cyclical Learning Rates for Training Neural Networks." IEEE Winter Conference on Applications of Computer Vision, 464-472. arXiv:1506.01186
8. ^Smith, L. N., and Topin, N. (2019). "Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates." Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, 11006, article 1100612. doi:10.1117/12.2520589
9. ^Loshchilov, I., and Hutter, F. (2017). "SGDR: Stochastic Gradient Descent with Warm Restarts." International Conference on Learning Representations. arXiv:1608.03983
10. ^Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). "Attention Is All You Need." Advances in Neural Information Processing Systems 30. NeurIPS
11. ^Goyal, P., Dollár, P., Girshick, R., et al. (2017). "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour." arXiv:1706.02677
12. ^Shallue, C. J., Lee, J., Antognini, J., et al. (2019). "Measuring the Effects of Data Parallelism on Neural Network Training." Journal of Machine Learning Research, 20(112), 1-49. JMLR
13. ^McCandlish, S., Kaplan, J., Amodei, D., and the OpenAI Dota Team. (2018). "An Empirical Model of Large-Batch Training." arXiv:1812.06162
14. ^Cohen, J. M., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. (2021). "Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability." International Conference on Learning Representations. OpenReview
15. ^Kalra, D. S., and Barkeshli, M. (2024). "Why Warmup the Learning Rate? Underlying Mechanisms and Improvements." Advances in Neural Information Processing Systems 37. NeurIPS
16. ^Defazio, A., Yang, X. A., Mehta, H., et al. (2024). "The Road Less Scheduled." Advances in Neural Information Processing Systems 37. NeurIPS
17. ^Schaipp, F., Hägele, A., Taylor, A., Simsekli, U., and Bach, F. (2025). "The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training." Proceedings of the 42nd International Conference on Machine Learning, 53267-53294. PMLR
18. ^Yang, G., Hu, E. J., Babuschkin, I., et al. (2021). "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer." Advances in Neural Information Processing Systems 34. NeurIPS
19. ^Howard, J., and Ruder, S. (2018). "Universal Language Model Fine-tuning for Text Classification." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 328-339. ACL Anthology
20. ^Bergstra, J., and Bengio, Y. (2012). "Random Search for Hyper-Parameter Optimization." Journal of Machine Learning Research, 13(10), 281-305. JMLR
21. ^Baydin, A. G., Cornish, R., Martínez-Rubio, D., Schmidt, M., and Wood, F. (2018). "Online Learning Rate Adaptation with Hypergradient Descent." International Conference on Learning Representations. OpenReview
22. ^Vaswani, S., Mishkin, A., Laradji, I., et al. (2019). "Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates." Advances in Neural Information Processing Systems 32. NeurIPS
23. ^Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2018). "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization." Journal of Machine Learning Research, 18(185), 1-52. JMLR