Gradient Descent

Gradient descent is an iterative first-order method for minimizing a differentiable scalar objective. Starting from a point, it evaluates the local gradient and moves in the opposite direction. The method is central to numerical optimization and to machine learning, where the objective is often a training loss function. Its guarantees are conditional: the step-size rule and properties of the objective determine whether the iterates decrease the objective, approach a minimizer, or only approach a stationary point.

The negative gradient is the direction of steepest local decrease when distance is measured with the Euclidean norm. A different norm or parameterization can produce a different steepest-descent direction, so "steepest" is not coordinate-free.^[4] Full gradient descent uses the exact gradient of the stated objective. Stochastic and mini-batch methods replace it with an estimate, trading deterministic progress per update for cheaper updates and access to large or streaming datasets.^[6]

Gradient descent is an optimizer, not the procedure that computes derivatives. In neural-network training, backpropagation or reverse-mode automatic differentiation computes a gradient, while gradient descent or another optimizer decides how to use it. Reducing training loss also does not by itself guarantee lower validation error or better generalization.

Basic formulation

For an unconstrained problem, let the objective be a differentiable function from an n-dimensional real vector space to the real numbers:

$$\min_{x \in \mathbb{R}^{n}} f(x).$$

At iteration k, ordinary gradient descent applies

$$x_{k+1} = x_k - \alpha_k \nabla f(x_k),$$

where alpha_k is a positive learning rate, also called a step size. The gradient collects the partial derivatives of f at x_k. For a small displacement d, the first-order approximation is

$$f(x_k+d) \approx f(x_k) + \nabla f(x_k)^{\mathsf T} d.$$

Among Euclidean unit directions, the negative normalized gradient minimizes the linear term. The unconstrained linear approximation has no finite minimum, however, so a step length or trust-region bound is essential.^[4]

Why a smoothness bound matters

A common assumption is that the gradient is L-Lipschitz:

$$\lVert \nabla f(x)-\nabla f(y) \rVert_2 \leq L\lVert x-y\rVert_2.$$

Under this assumption, the descent lemma gives

$$f(x_{k+1}) \leq f(x_k) - \alpha_k \left(1-\frac{L\alpha_k}{2}\right) \lVert \nabla f(x_k)\rVert_2^2.$$

Thus an exact gradient step decreases the objective whenever the gradient is nonzero and 0 < alpha_k < 2/L. The conservative choice alpha_k <= 1/L is common in convergence proofs.^[5][6] This result does not imply that a stochastic mini-batch loss decreases on every update, and it does not apply without a suitable smoothness condition.

Minimal algorithm

An implementation of basic gradient descent has four decisions:

1. Choose an initial point x_0.
2. Evaluate g_k = grad f(x_k).
3. Select a positive step size alpha_k and set x_{k+1} = x_k - alpha_k g_k.
4. Stop according to a stated criterion, such as a small gradient norm, a small relative change, a step or evaluation budget, or lack of improvement in a separate validation metric.

A small gradient norm is a first-order stationarity test. It is not proof of global optimality for a nonconvex objective. A small change in the objective is also not sufficient by itself: progress can appear small because the learning rate is too small or the problem is poorly scaled.

Step-size selection

The step size controls both stability and speed. No single numerical value is meaningful across arbitrary objectives because rescaling the objective or its parameters rescales the gradient.

Fixed steps

For an L-smooth objective, a fixed step at or below 1/L supports standard deterministic convergence bounds. In a convex quadratic, the usable range depends on the Hessian eigenvalues. Directions with high curvature limit the stable step, while low-curvature directions may then make slow progress. This is one reason a poorly conditioned problem causes the familiar zigzag trajectory.^[4][5]

Line search

A line search chooses a step after examining the objective along a descent direction. Exact line search minimizes that one-dimensional function, but the extra function evaluations may be costly. Backtracking instead starts from a trial step and shrinks it until a sufficient-decrease condition holds. Line search is standard in numerical optimization, but it is less common for large mini-batch neural-network training because a noisy batch objective is not a stable proxy for the full objective and repeated evaluations can be expensive.^[4][6]

Diminishing and scheduled steps

Classical stochastic approximation commonly assumes a diminishing sequence satisfying

$$\sum_{k=0}^{\infty}\alpha_k = \infty \quad\text{and}\quad \sum_{k=0}^{\infty}\alpha_k^2 < \infty.$$

These conditions prevent the method from exhausting its total travel while making the cumulative effect of persistent finite-variance noise controllable. They are not a universal prescription for finite-horizon deep-learning runs; the accompanying assumptions about sampling, noise, and the objective matter.^[3][6]

Practical training schedules include piecewise-constant decay, exponential decay, polynomial decay, and cosine annealing. SGDR introduced cosine cycles with warm restarts as an empirical training schedule.^[18] Warmup gradually increases the step size at the start of training. Goyal and colleagues used gradual warmup together with a linear learning-rate scaling rule for a specific large-batch ImageNet experiment, but that result is an empirical recipe for that setting rather than a theorem that every model should use the same rule.^[19]

Convergence guarantees

Convergence statements must name the objective class, the step-size rule, the output being measured, and whether the gradient is exact or stochastic. The following are standard deterministic results for an L-smooth objective and a step of 1/L.^[5]

Scroll sideways for more →

For the nonconvex row, an O(1/k) bound on the squared gradient norm means that obtaining a gradient norm at most epsilon generally requires O(1/epsilon^2) iterations in this basic analysis. Stating O(1/epsilon) without saying that epsilon measures the squared gradient norm changes the claim.

Convexity

If f is convex, every local minimizer is global. With smoothness and an appropriate fixed step, basic gradient descent approaches a global minimizer in objective value. Strong convexity adds a quadratic growth condition and yields a geometric rate. The ratio L/mu is a condition number in this analysis: a larger ratio means that the guaranteed contraction is closer to one and therefore slower.^[5]

Convexity alone is not enough for the smooth-gradient update and rate above. A convex but nondifferentiable objective calls for a subgradient or proximal method, and its rates and stopping tests differ.

Nonconvex objectives

For a smooth nonconvex optimization problem, a zero gradient can occur at a local minimum, a local maximum, or a saddle point. The generic first-order result is therefore about stationarity, not global optimality.^[6] Random initialization and a sufficiently short fixed step avoid convergence to strict saddles under the regularity assumptions analyzed by Lee and colleagues, but this does not cover every degenerate saddle or guarantee a global minimum.^[26] Perturbed gradient methods add controlled noise and have stronger escape results under additional smoothness and strict-saddle conditions.^[27]

The distinction matters for neural networks. Their objectives can have symmetries, flat directions, nonsmooth points, and many equivalent parameterizations. A convergence theorem for smooth finite-dimensional functions should not be quoted as if it automatically characterized every training run.

Stochastic rates

Stochastic-gradient rates depend on convexity, gradient-noise assumptions, step sizes, averaging, and whether the result measures objective error, distance, regret, or gradient norm. Under standard bounded-variance assumptions, noise can impose a floor for a constant learning rate. Diminishing steps or variance-reduction methods can remove that floor in settings covered by their analyses.^[6] A single rate such as O(1/sqrt(k)) is therefore not a complete description of stochastic gradient descent.

Full, stochastic, and mini-batch gradients

Many supervised-learning objectives are finite sums:

$$F(x) = \frac{1}{N}\sum_{i=1}^{N} f_i(x).$$

The three common data regimes differ in how they estimate the gradient.

Full-batch gradient descent

Full-batch gradient descent evaluates all N component gradients before each update:

$$g_k = \nabla F(x_k).$$

For a fixed dataset and deterministic arithmetic, this is the exact gradient of the empirical objective. Its update is comparatively expensive when N is large, but it fits the deterministic convergence analysis directly.

Single-example stochastic gradient descent

Stochastic gradient descent samples an index i_k and uses

$$g_k = \nabla f_{i_k}(x_k).$$

If indices are sampled uniformly and independently, this estimator is unbiased conditional on the current iterate:

$$\mathbb{E}[g_k \mid x_k] = \nabla F(x_k).$$

Unbiasedness does not mean that each update decreases F. Individual gradients can point uphill for the full objective, and successive estimates can be dependent when data are shuffled without replacement.

Mini-batch stochastic gradient descent

For a mini-batch B_k of size b,

$$g_k = \frac{1}{b} \sum_{i\in B_k} \nabla f_i(x_k).$$

Under independent sampling, averaging reduces estimator variance as the batch grows, but the benefit is workload-dependent and eventually limited by correlation, finite data, optimization dynamics, and available parallelism. The batch size also changes how many parameter updates occur for a fixed number of examples. Comparisons must therefore state whether they hold epochs, updates, examples, or compute fixed.

Mini-batches can improve accelerator utilization and data-parallel throughput. They do not provide unlimited speedup. A large empirical study by Shallue and colleagues found substantial variation in the useful range of data parallelism across workloads and found that disagreements about large-batch generalization could often be explained by tuning and budget differences.^[23] This is stronger evidence than a blanket rule that large batches necessarily find worse solutions.

Gradient accumulation and data parallelism

Gradient accumulation sums or averages gradients from several micro-batches before one optimizer update. Synchronous data parallelism similarly combines gradients produced by multiple workers. If loss reduction, gradient reduction, and learning-rate scaling are consistent, these can reproduce a larger effective mini-batch. They are not equivalent when workers take local optimizer steps, use different data, apply clipping before rather than after reduction, or normalize gradients differently.

For reproducibility, report the per-device micro-batch, number of workers, accumulation steps, global batch, whether gradients are summed or averaged, and how the learning rate changed.

Momentum and acceleration

Momentum adds state that combines current and past gradients. It can damp oscillation across high-curvature directions and build motion along a direction that remains consistent.

Heavy-ball momentum

A common convention for the heavy-ball method is

$$v_{k+1} = \beta v_k + \nabla f(x_k),$$ $$x_{k+1} = x_k - \alpha v_{k+1}.$$

Other sources put the learning rate inside the velocity update. Those conventions are equivalent only after translating the state and hyperparameters, so optimizer settings should not be copied across formulas without checking the implementation. Polyak introduced the heavy-ball acceleration idea in 1964.^[7] Its strongest classical acceleration results require more structure than arbitrary nonconvex neural-network losses.

Nesterov acceleration

Nesterov's accelerated method evaluates a gradient at an extrapolated point and couples that step to a carefully chosen sequence of extrapolation coefficients. For smooth convex optimization, it attains the O(1/k^2) worst-case objective rate established in the original 1983 work and modern treatments.^[5][8]

Deep-learning libraries also expose options named "Nesterov momentum." Their velocity notation and finite-horizon behavior can differ from the canonical convex accelerated-gradient algorithm. The convex O(1/k^2) theorem should be attributed to the full accelerated scheme and its assumptions, not to every lookahead-style momentum implementation. Sutskever and colleagues studied a particular momentum reformulation and schedule empirically for deep and recurrent networks.^[11]

Adaptive diagonal methods

Adaptive methods maintain coordinate-wise statistics of past gradients. They can be viewed as using a time-varying diagonal preconditioner rather than one scalar step for all coordinates. Their behavior depends on initialization, epsilon placement, bias correction, and update ordering, which differ among implementations.

AdaGrad

The diagonal AdaGrad accumulator is

$$r_k = r_{k-1} + g_k \odot g_k,$$

with update

$$x_{k+1} = x_k - \alpha \frac{g_k}{\sqrt{r_k}+\epsilon},$$

where multiplication, division, and the square root are coordinate-wise. The original work develops adaptive subgradient methods with regret guarantees and highlights their response to sparse, infrequently observed features.^[12] Because r_k never decreases, the effective coordinate steps can become very small in a long run.

RMSProp

RMSProp replaces AdaGrad's cumulative sum with an exponential moving average:

$$r_k = \rho r_{k-1} + (1-\rho)g_k\odot g_k,$$ $$x_{k+1} = x_k - \alpha \frac{g_k}{\sqrt{r_k+\epsilon}}.$$

The method is documented in Geoffrey Hinton's 2012 neural-network course notes.^[13] Libraries vary in whether epsilon is inside or outside the square root and whether momentum or centering is added. Those are different update rules, not merely formatting choices.

Adam and AMSGrad

Adam combines an exponential average of gradients with an exponential average of squared gradients:

$$m_k = \beta_1m_{k-1} + (1-\beta_1)g_k,$$ $$v_k = \beta_2v_{k-1} + (1-\beta_2)g_k\odot g_k.$$

With zero-initialized moments, the original algorithm applies bias corrections

$$\widehat m_k = \frac{m_k}{1-\beta_1^k}, \qquad \widehat v_k = \frac{v_k}{1-\beta_2^k},$$

then updates

$$x_{k+1} = x_k - \alpha \frac{\widehat m_k}{\sqrt{\widehat v_k}+\epsilon}.$$

Kingma and Ba proposed default values beta_1 = 0.9, beta_2 = 0.999, and epsilon = 10^{-8} for the original algorithm, while emphasizing that the learning rate is problem-dependent.^[14] These are defaults from one specification, not universal optima.

The original convergence proof did not cover all parameter settings later used in practice. Reddi, Kale, and Kumar constructed a simple convex online example in which Adam fails to converge and proposed AMSGrad, which retains a coordinate-wise maximum of past second-moment estimates.^[15] AMSGrad has convergence guarantees under stated assumptions; it is not a proof that every Adam-family implementation converges on every nonconvex problem.

AdamW and weight decay

AdamW decouples parameter decay from Adam's loss-gradient preconditioner. In schematic form,

$$x_{k+1} = (1-\alpha\lambda)x_k - \alpha \frac{\widehat m_k}{\sqrt{\widehat v_k}+\epsilon}.$$

For plain stochastic gradient descent, an L2 penalty and multiplicative weight decay can be made equivalent after accounting for the learning rate. Loshchilov and Hutter showed that this equivalence does not hold for adaptive scaling, because adding the L2 derivative to the loss gradient also sends it through the adaptive preconditioner.^[16] Decoupling fixes that specific mismatch. It does not establish that AdamW is best for every architecture or dataset.

Lion

Lion uses a sign-based update built from a momentum-like state and keeps one such state tensor rather than Adam's two moment tensors. It was discovered through a symbolic program search and evaluated on selected vision, language, and diffusion workloads.^[17] The paper reports competitive results and lower optimizer-state memory than Adam, but it also documents workloads where gains were small or not statistically significant. Lion is therefore an evaluated alternative, not a general dominance result.

Comparison

Scroll sideways for more →

State counts alone do not determine total training memory. Master weights, gradient buffers, low-precision copies, sharding, offloading, and framework-specific temporary storage must also be included.

Choosing and comparing a method

Optimizer choice should follow the structure of the problem and the comparison budget. For a moderate deterministic problem where objective and gradient evaluations are reliable, basic gradient descent with backtracking is a useful baseline, but a quasi-Newton method may need far fewer iterations. For a finite sum too large to evaluate on every update, a stochastic or mini-batch method usually has a lower cost per update. Sparse features can make AdaGrad's coordinate adaptation useful, while a dense neural network may be evaluated with momentum SGD, AdamW, or another method. These are starting points for measurement, not rules that establish a winner in advance.^[6][28]

A fair comparison separates at least four budgets:

• Update budget: the same number of parameter updates.
• Example budget: the same number of training examples or tokens processed.
• Compute budget: comparable arithmetic work or accelerator time.
• Tuning budget: comparable opportunity to choose learning rates, schedules, decay, and other hyperparameters.

Equal update counts can favor a method using a larger batch because it sees more examples. Equal epochs can favor a method with cheaper but more numerous updates. Equal wall-clock time can reflect implementation maturity or hardware utilization as much as mathematical progress. Reporting more than one axis, such as validation metric against both examples and elapsed time, makes the tradeoff visible.

The learning rate should be tuned for each optimizer rather than copied across differently scaled updates. Decoupled weight decay, clipping thresholds, and schedules may also need separate tuning. If an optimizer uses more state, include its memory traffic and any reduction in feasible batch size. If it reaches a target faster but a different final metric, report both time-to-target and the fixed-budget result.

At minimum, compare against a simple well-tuned baseline and include variability across seeds when randomness is material. An isolated best run cannot distinguish a reliable improvement from favorable initialization or data order. The correct practical conclusion may be conditional, such as one method reaching a target sooner at a higher state-memory cost, rather than that one optimizer is universally superior.

Neural-network training

For a neural network, the empirical objective is commonly an average of per-example losses plus any explicit regularization. Reverse-mode automatic differentiation efficiently evaluates derivatives of a scalar output with respect to many parameters. Backpropagation is the neural-network application of that broader technique; it is not itself an optimizer.^[9][10]

Conditioning and parameter scaling

Gradient descent is not invariant to arbitrary rescaling of coordinates. If one parameter changes the objective sharply and another changes it weakly, a scalar step that is safe for the first can be slow for the second. Feature normalization, suitable parameterization, initialization, normalization layers, and preconditioning can change the effective conditioning. Adaptive methods provide diagonal scaling, while Newton and quasi-Newton methods use richer curvature approximations at greater computational and memory cost.

Monitoring only the scalar loss can conceal this issue. Useful diagnostics include gradient norms by layer, update-to-parameter ratios, activation and gradient distributions, and the fraction of values that are nonfinite or clipped.

Gradient clipping

Gradient clipping modifies a computed gradient before the optimizer update. Global norm clipping uses

$$\widetilde g = g \min \left( 1, \frac{c}{\lVert g\rVert_2} \right),$$

where c is the threshold. It preserves direction when clipping occurs and caps the norm. Coordinate-wise value clipping is a different operation. Pascanu, Mikolov, and Bengio proposed norm clipping as a response to exploding gradients in recurrent-network experiments.^[20]

Clipping bounds an update input; it does not repair the source of a nonfinite gradient, guarantee convergence, or make any threshold universally correct. Its placement also matters in distributed training. Clipping each worker before averaging is not generally equivalent to clipping the aggregated gradient.

Regularization and early stopping

Explicit regularization changes the training objective or procedure. Examples include an L2 penalty, decoupled weight decay, data augmentation, dropout, and early stopping. These must be distinguished from an optimizer's implicit bias. A lower training objective can coexist with a higher validation error, so validation-based model selection is separate from the optimizer's stationarity test.

Mixed precision

Mixed-precision training performs selected operations and stores selected tensors in lower-precision formats while retaining higher precision where needed. The FP16 method studied by Micikevicius and colleagues used an FP32 master copy of weights, loss scaling to protect small gradients, and FP32 accumulation for some operations.^[21] BF16 has the same exponent width as FP32 and therefore a much wider dynamic range than IEEE FP16, although it has fewer fraction bits.^[22]

The phrase "mixed precision" does not specify a unique numerical algorithm. A reproducible report should identify the formats used for parameters, forward activations, backward gradients, reductions, optimizer states, and accumulators, as well as the loss-scaling policy. Reduced precision can change rounding, underflow, overflow, and distributed-reduction behavior.

Generalization and implicit bias

Optimization and generalization answer different questions. Optimization asks how well a procedure reduces the chosen training objective. Generalization concerns performance on data outside that finite training sample. An optimizer can reach a lower training loss but a worse validation metric.

A precise implicit-bias result

When an underdetermined problem has many solutions, the update rule and initialization can favor some solutions without an explicit penalty. This is called implicit bias. One rigorous example is unregularized logistic regression with homogeneous linear predictors on linearly separable data: under the assumptions in Soudry and colleagues' analysis, the direction of gradient-descent iterates converges to the hard-margin support-vector-machine direction even though the parameter norm diverges.^[24]

That result is specific. It does not prove that gradient descent always finds a low-norm, maximum-margin, or otherwise "simple" solution in an arbitrary deep network. Parameterization, architecture, loss, step size, stochasticity, and stopping time can all change the bias.

Flatness claims require a definition

Small-batch noise and the geometry around a solution are often discussed as explanations for generalization. A raw claim that "flat minima generalize and sharp minima do not" is not invariant to parameterization. Dinh and colleagues showed that functionally equivalent rectified networks can be reparameterized to make commonly used sharpness measures arbitrarily different without changing the represented function or its generalization.^[25]

Flatness can still be useful when a measure, neighborhood, normalization, and parameterization are defined, but it should not be treated as an optimizer-independent scalar fact. Likewise, evidence about batch size must control the learning-rate schedule, tuning budget, number of updates, examples processed, and compute budget.^[23]

Failure modes and diagnosis

Divergence or oscillation

An excessively large learning rate can overshoot or produce an unstable recurrence, especially along high-curvature directions. Momentum can amplify the instability. Useful checks are to lower the step, inspect the first failing update, verify loss normalization and gradient reduction, and distinguish a finite but growing loss from the first NaN or infinity. Clipping may limit a finite spike but should not be used to hide a systematic numerical error.

Very slow progress

A learning rate that is too small produces little movement. Poor conditioning can produce slow progress even at the largest stable scalar step. Saturated activations, badly scaled features, and an unsuitable parameterization can also yield small or unbalanced gradients. Examine gradients and updates by parameter group rather than inferring the cause from loss alone.

A noisy or rising mini-batch loss

A stochastic update need not reduce the next mini-batch loss or the full objective. Compare consistently defined averages, and separate sampling variation from a sustained trend. If increasing the batch changes the number of optimizer steps, learning-rate schedule, or normalization, it is not an isolated batch-size experiment.

Training improves while validation worsens

This is not an optimizer-convergence contradiction. The training objective and out-of-sample metric differ. Review data splits, leakage, distribution shift, capacity, explicit regularization, and stopping time. Overfitting is a statistical issue even when the optimizer behaves exactly as specified.

Distributed runs disagree

Check whether gradients are summed or averaged, where clipping occurs, whether all workers use the same parameters before an update, and how dropped or uneven batches are handled. Floating-point reductions are order-dependent, so bitwise identity is not always expected even when runs are mathematically intended to match.

Adaptive optimizer mismatch

Confirm the exact update rather than relying on its name. Epsilon placement, bias correction, coupled versus decoupled decay, momentum convention, sparse-gradient handling, and parameter exclusions can materially change a run. Loading optimizer state with a different implementation can also change the recurrence.

Boundaries with related methods

Gradient descent in its basic form assumes an unconstrained differentiable objective and uses first derivatives.

• Projected gradient descent takes a gradient step and then projects onto a feasible set.
• A subgradient method handles certain nondifferentiable convex objectives but does not inherit smooth gradient descent's rates.
• A proximal-gradient method treats a sum of a smooth term and a simple possibly nondifferentiable term by combining a gradient step with a proximal operator.
• Newton's method uses a Hessian matrix, and quasi-Newton methods approximate curvature. Their steps can be more informative but more expensive than a gradient.
• Coordinate descent updates selected variables rather than the full gradient direction.
• Mirror descent changes the geometry of the step and can be better matched to non-Euclidean domains.

Calling all of these "gradient descent variants" can obscure different assumptions and guarantees. The precise update and geometry should be stated.

History

Augustin-Louis Cauchy's 1847 note described a descent procedure in the course of solving simultaneous equations by reducing a sum of squares.^[1] Haskell Curry published an early convergence study of steepest descent for nonlinear minimization in 1944.^[2] Robbins and Monro's 1951 stochastic-approximation method established the foundation for recursive procedures driven by noisy observations.^[3]

Polyak introduced the heavy-ball acceleration method in 1964.^[7] Nesterov's 1983 paper gave an accelerated method with an O(1/k^2) rate for smooth convex optimization.^[8] Rumelhart, Hinton, and Williams' 1986 paper helped popularize training multilayer neural networks by backpropagating error derivatives.^[9] Reverse-mode differentiation has a broader history and scope than that neural-network paper, as documented in the automatic-differentiation literature.^[10]

Adaptive first-order methods became prominent in machine learning with AdaGrad in 2011, RMSProp course notes in 2012, and Adam released in 2014 and presented at ICLR 2015.^[12][13][14] Subsequent work identified limits of Adam's original convergence argument, separated weight decay from adaptive preconditioning in AdamW, and explored alternatives such as Lion.^[15][16][17]

Evaluation and reporting checklist

A technically meaningful gradient-based optimization report should include:

• The exact objective, reduction convention, and any regularization terms.
• The optimizer equations or an implementation and version precise enough to recover them.
• Initialization, scalar and parameter-group learning rates, momentum or moment coefficients, epsilon, decay, clipping, and schedule.
• Full-batch, sampling, shuffle, replacement, and mini-batch details.
• Micro-batch, accumulation, worker count, global batch, and gradient-reduction semantics.
• Numerical formats, loss scaling, master-weight and optimizer-state precision, and nonfinite-value handling.
• The stopping rule and whether results are selected by training objective, validation metric, or a fixed budget.
• At least one optimization diagnostic, such as objective evaluations, gradient norm, update norm, or time to a stated target.
• Random seeds and variability across runs when stochastic effects are material.

Without these details, two runs both labeled "AdamW" or "SGD with momentum" may implement meaningfully different recurrences.

References

1. ^Cauchy, A.-L. (1847). "Methode generale pour la resolution des systemes d'equations simultanees." *Comptes rendus hebdomadaires des seances de l'Academie des sciences*, 25, 536-538. gallica.bnf.fr/...f540
2. ^Curry, H. B. (1944). "The Method of Steepest Descent for Non-linear Minimization Problems." *Quarterly of Applied Mathematics*, 2(3), 258-261. doi.org/...10667
3. ^Robbins, H., and Monro, S. (1951). "A Stochastic Approximation Method." *The Annals of Mathematical Statistics*, 22(3), 400-407. doi.org/...1177729586
4. ^Boyd, S., and Vandenberghe, L. (2004). *Convex Optimization*, Chapter 9. Cambridge University Press. web.stanford.edu/...bv_cvxbook.pdf
5. ^Bubeck, S. (2015). "Convex Optimization: Algorithms and Complexity." *Foundations and Trends in Machine Learning*, 8(3-4), 231-357. arxiv.org/...1405.4980
6. ^Bottou, L., Curtis, F. E., and Nocedal, J. (2018). "Optimization Methods for Large-Scale Machine Learning." *SIAM Review*, 60(2), 223-311. doi.org/...16M1080173
7. ^Polyak, B. T. (1964). "Some methods of speeding up the convergence of iteration methods." *USSR Computational Mathematics and Mathematical Physics*, 4(5), 1-17. doi.org/...0041-5553(64)90137-5
8. ^Nesterov, Y. E. (1983). "A method of solving a convex programming problem with convergence rate O(1/k^2)." *Doklady Akademii Nauk SSSR*, 269(3), 543-547. mathnet.ru/...dan46009
9. ^Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). "Learning representations by back-propagating errors." *Nature*, 323, 533-536. doi.org/...323533a0
10. ^Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. (2018). "Automatic Differentiation in Machine Learning: a Survey." *Journal of Machine Learning Research*, 18(153), 1-43. jmlr.org/...17-468
11. ^Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013). "On the importance of initialization and momentum in deep learning." *Proceedings of Machine Learning Research*, 28(3), 1139-1147. proceedings.mlr.press/...sutskever13
12. ^Duchi, J., Hazan, E., and Singer, Y. (2011). "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization." *Journal of Machine Learning Research*, 12(61), 2121-2159. jmlr.org/...duchi11a
13. ^Tieleman, T., and Hinton, G. (2012). "Lecture 6e: RMSProp: Divide the gradient by a running average of its recent magnitude." *Neural Networks for Machine Learning*. cs.toronto.edu/...lec6.pdf
14. ^Kingma, D. P., and Ba, J. (2015). "Adam: A Method for Stochastic Optimization." *International Conference on Learning Representations*. arxiv.org/...1412.6980
15. ^Reddi, S. J., Kale, S., and Kumar, S. (2018). "On the Convergence of Adam and Beyond." *International Conference on Learning Representations*. openreview.net/forum
16. ^Loshchilov, I., and Hutter, F. (2019). "Decoupled Weight Decay Regularization." *International Conference on Learning Representations*. openreview.net/forum
17. ^Chen, X., Liang, C., Huang, D., Real, E., et al. (2023). "Symbolic Discovery of Optimization Algorithms." *Advances in Neural Information Processing Systems*, 36. proceedings.neurips.cc/...ccba8757137d84f-Abstract
18. ^Loshchilov, I., and Hutter, F. (2017). "SGDR: Stochastic Gradient Descent with Warm Restarts." *International Conference on Learning Representations*. openreview.net/forum
19. ^Goyal, P., Dollar, P., Girshick, R., Noordhuis, P., et al. (2017). "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour." arxiv.org/...1706.02677
20. ^Pascanu, R., Mikolov, T., and Bengio, Y. (2013). "On the difficulty of training recurrent neural networks." *Proceedings of Machine Learning Research*, 28(3), 1310-1318. proceedings.mlr.press/...pascanu13
21. ^Micikevicius, P., Narang, S., Alben, J., Diamos, G., et al. (2018). "Mixed Precision Training." *International Conference on Learning Representations*. openreview.net/forum
22. ^Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., et al. (2019). "A Study of BFLOAT16 for Deep Learning Training." arxiv.org/...1905.12322
23. ^Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G. E. (2019). "Measuring the Effects of Data Parallelism on Neural Network Training." *Journal of Machine Learning Research*, 20(112), 1-49. jmlr.org/...18-789
24. ^Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. (2018). "The Implicit Bias of Gradient Descent on Separable Data." *Journal of Machine Learning Research*, 19(70), 1-57. jmlr.org/...18-188
25. ^Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. (2017). "Sharp Minima Can Generalize For Deep Nets." *Proceedings of Machine Learning Research*, 70, 1019-1028. proceedings.mlr.press/...dinh17b
26. ^Lee, J. D., Simchowitz, M., Jordan, M. I., and Recht, B. (2016). "Gradient Descent Only Converges to Minimizers." *Proceedings of Machine Learning Research*, 49, 1246-1257. proceedings.mlr.press/...lee16
27. ^Jin, C., Ge, R., Netrapalli, P., Kakade, S. M., and Jordan, M. I. (2017). "How to Escape Saddle Points Efficiently." *Proceedings of Machine Learning Research*, 70, 1724-1732. proceedings.mlr.press/...jin17a
28. ^Goodfellow, I., Bengio, Y., and Courville, A. (2016). "Optimization for Training Deep Models." In *Deep Learning*, Chapter 8. MIT Press. deeplearningbook.org/...optimization