Stochastic Gradient Descent (SGD)

Stochastic gradient descent (SGD) is a first-order optimization method that updates parameters using a gradient estimate computed from a randomly selected example or subset of examples. It is a stochastic counterpart of gradient descent, which uses the gradient of the complete objective at every update. In machine learning, the objective is often an average over a training set or an expectation over a data distribution, so a small sample can provide a much cheaper update than a full gradient.^[1][2]

The name covers several related procedures. In theoretical work, "SGD" often means sampling one example independently with replacement at every iteration. In deep learning, it commonly means mini-batch SGD, often with momentum, where each update averages gradients over a batch. Many training programs instead shuffle the dataset and process it without replacement for one epoch. That procedure is usually called random reshuffling in optimization theory. Its gradients are dependent within an epoch, so proofs for independent sampling do not automatically apply.^[2][19][20]

SGD is an optimization algorithm, not a guarantee of statistical accuracy or a single recipe for training every model. Its behavior depends on the objective, sampling rule, learning rate, batch size, momentum, stopping rule, and assumptions used in the analysis. For convex objectives, suitably configured SGD can approach a global minimizer. For general nonconvex objectives, including those of many neural networks, standard theory normally establishes convergence toward approximate stationary points, not the global minimum.^[2][3]

Optimization settings

Expected objectives

A stochastic optimization problem can be written as

$$F(\theta) = \mathbb{E}_{\xi \sim P}[f(\theta;\xi)],$$

where $\theta$ is the parameter vector, $\xi$ is a random observation, and $f(\theta;\xi)$ is the loss associated with that observation. The goal is to minimize the population objective $F$, but its exact expectation may be unavailable or too expensive to calculate. A stochastic oracle instead returns a vector $g(\theta,\xi)$ computed from a sampled $\xi$.^[2][3]

Classical analyses often assume that the oracle is conditionally unbiased:

$$\mathbb{E}[g(\theta_t,\xi_t)\mid \theta_t] = \nabla F(\theta_t).$$

They may also assume bounded variance or a bounded second moment. These are assumptions of a particular theorem, not properties that every practical gradient estimator has. Biased sampling, data dependence, clipping, quantization, stale parameters, or a stateful model can change the estimator and require a different analysis.^[2][3]

Finite-sum objectives

For a fixed training set with $n$ examples, empirical risk minimization commonly produces a finite-sum objective

$$F_n(\theta) = \frac{1}{n}\sum_{i=1}^{n} f_i(\theta).$$

A full-gradient update evaluates every component:

$$\theta_{t+1} = \theta_t - \alpha_t \nabla F_n(\theta_t).$$

An SGD update samples an index $i_t$ uniformly with replacement and uses

$$\theta_{t+1} = \theta_t - \alpha_t \nabla f_{i_t}(\theta_t).$$

Conditioned on $\theta_t$, uniform with-replacement sampling makes $\nabla f_{i_t}(\theta_t)$ an unbiased estimator of $\nabla F_n(\theta_t)$. A mini-batch estimate averages several sampled component gradients. If those samples are independent and have finite variance, averaging reduces the estimator variance, although it also increases the work per update.^[2]

A regularizer can be included in the objective. If

$$\tilde F_n(\theta) = \frac{1}{n}\sum_{i=1}^{n} f_i(\theta) + \lambda R(\theta),$$

then a differentiable regularizer contributes $\lambda\nabla R(\theta_t)$ to each update. A nonsmooth regularizer may instead call for a proximal stochastic method. Such methods are related to SGD, but the projection or proximal operation is part of the algorithm and of its convergence assumptions.^[4][23][24]

Online and finite-data interpretations

The expected-risk and finite-sum views are related but not interchangeable. In a streaming problem, a new independent observation can be drawn at each step. In a finite dataset, examples are reused. Reuse changes the distinction between optimization error on the training objective and generalization to new data. It also creates choices about replacement, ordering, and the meaning of an epoch.^[2][8]

An epoch is one dataset-equivalent amount of example processing. With single-example updates, one epoch contains $n$ component-gradient evaluations. With a batch of $b$ examples, it contains about $n/b$ updates if every example is used once. Counting epochs, updates, component-gradient evaluations, and elapsed time answers different questions, so comparisons should state which unit is used.^[2][15]

Basic algorithm

For a mini-batch $B_t$ and learning rate $\alpha_t$, the basic update is

$$g_t = \frac{1}{|B_t|}\sum_{i\in B_t}\nabla f_i(\theta_t),$$ $$\theta_{t+1} = \theta_t - \alpha_t g_t.$$

A minimal training loop is:

choose initial parameters theta
for t = 0, 1, 2, ...:
    select a sample or mini-batch B_t
    compute the mean gradient g_t at theta
    choose the learning rate alpha_t
    theta = theta - alpha_t * g_t
return a specified iterate or an average of iterates

This pseudocode leaves important choices explicit. It does not prescribe how to initialize the model, how to sample data, whether to use momentum, when to change the learning rate, or which iterate to return. Those decisions can alter both the method and the applicable theory.^[2][4][6]

Full batch, single sample, and mini-batch

Scroll sideways for more →

The table assumes the same parameter vector is used for all component gradients in an update. Splitting a batch into micro-batches and accumulating their summed gradients can reproduce a large-batch gradient when the loss scaling and model computation are equivalent. Updating parameters after every micro-batch is a different sequence of SGD steps.^[2]

The largest batch that fits in memory is not automatically the best batch. Larger batches can reduce the number of training steps and expose more hardware parallelism, but their marginal reduction in required steps eventually diminishes in many workloads. The location of that transition varies considerably with the model, dataset, optimizer, error target, and tuning procedure.^[15][16]

Historical background

Robbins and Monro introduced stochastic approximation in 1951 as a sequential method for finding the root of an unknown regression function from noisy observations. Their procedure was not presented as modern neural-network training, but it supplied a foundational framework for iterative updates driven by noisy information.^[1]

The stochastic-gradient method inherits that framework when the noisy observation is a gradient estimate. Bottou, Curtis, and Nocedal describe stochastic gradients as central to large-scale machine learning because an update can use a small amount of data rather than repeatedly scanning the entire training set.^[2]

Momentum has a separate history. Polyak's 1964 work studied multistep iterative methods for accelerating deterministic optimization. Later neural-network work combined related momentum recurrences with stochastic gradients. Sutskever and colleagues showed in 2013 that carefully selected initialization and momentum schedules could train the deep and recurrent networks used in their experiments competitively with a more complex Hessian-free method. That result concerned the tested architectures and tasks; it did not establish momentum as universally superior.^[5][6]

Learning rates

The learning rate controls the size of every update. A value that is too large can make the iterates unstable or keep them far from a solution. A value that is too small can make useful progress unreasonably slow. The acceptable scale depends on curvature, gradient noise, parameterization, batching, and any momentum convention.^[2][6]

Constant and diminishing sequences

Under classical stochastic-approximation assumptions, a common sufficient condition for almost-sure convergence uses positive learning rates satisfying

$$\sum_{t=1}^{\infty}\alpha_t = \infty \quad\text{and}\quad \sum_{t=1}^{\infty}\alpha_t^2 < \infty.$$

The first condition prevents the total movement from becoming finite too early. The second limits the accumulated effect of persistent noise. A power-law sequence $\alpha_t=c/t^p$ satisfies both when $1/2 < p \leq 1$. In particular, $\alpha_t=c/\sqrt{t}$ does not satisfy the square-summability condition because its squared terms form the divergent harmonic series. These conditions are not necessary for every SGD variant, and they do not replace the other assumptions in a convergence theorem.^[1][2]

With a fixed positive learning rate and nonzero gradient variance, strongly convex analyses generally show a contracting transient followed by a noise-dependent neighborhood of the optimum. Reducing the fixed learning rate shrinks that neighborhood but slows the transient. A suitable diminishing sequence can remove the asymptotic noise floor, at the cost of progressively smaller updates.^[2]

Scheduled learning rates

Practical schedules include piecewise reductions, exponential decay, polynomial decay, and cosine-shaped cycles. They are design choices rather than consequences of one universal theorem. Loshchilov and Hutter's SGDR method used cosine annealing with warm restarts and reported empirical improvements on CIFAR-10, CIFAR-100, an EEG dataset, and a downsampled ImageNet setting. A warm restart raises the learning rate at a cycle boundary; it does not restore the model parameters to their initial values.^[7]

Warmup starts with a smaller learning rate and increases it during early training. In Goyal and colleagues' ResNet-50 ImageNet experiments, gradual warmup was part of the recipe that made a linearly scaled learning rate work at a global mini-batch of 8,192. That is evidence for one large-batch image-classification setting, not a general proof that linear scaling or warmup is optimal for every model.^[14]

A schedule should be specified in units that survive changes to batch size and parallelism. "Decay after 10,000 steps" and "decay after 10 epochs" are different when the number of updates per epoch changes. Reproducible reports state the base learning rate, schedule, warmup, batch size, momentum convention, and total update or data budget.^[14][15]

Momentum and iterate averaging

Classical momentum

One common momentum convention maintains a velocity:

$$v_{t+1} = \mu v_t - \alpha_t g_t,$$ $$\theta_{t+1} = \theta_t + v_{t+1},$$

where $\mu$ is the momentum coefficient. Equivalent-looking implementations may place the learning rate, dampening, or weight decay in different parts of the recurrence. Two systems that use the same displayed value of $\mu$ can therefore implement different updates.^[5][6]

Momentum accumulates information from previous gradients. On objectives with directions of very different curvature, it can preserve motion in directions whose gradients remain consistent while damping some oscillation in directions that change sign. This intuition does not by itself establish a convergence rate for a nonconvex stochastic objective. Formal guarantees depend on the exact recurrence and assumptions, and empirical gains depend on tuning.^[5][6]

Nesterov-style momentum evaluates or combines gradient information differently from classical heavy-ball momentum. Sutskever and colleagues analyzed the local difference and compared both in their deep-learning experiments. It is inaccurate to treat every optimizer labeled "Nesterov" as byte-for-byte identical without checking its implementation.^[6]

Averaging model iterates

Iterate averaging is distinct from momentum. An averaged output can be written

$$\bar\theta_T = \sum_{t=1}^{T} w_t\theta_t, \qquad \sum_{t=1}^{T}w_t=1.$$

Uniform averaging, suffix averaging, and polynomial-decay averaging use different weights. For nonsmooth convex stochastic optimization, Shamir and Zhang proved optimal rates for particular averaging schemes and showed that the last iterate can have additional logarithmic factors. Their results require convexity, bounded stochastic subgradients, and the stated learning-rate choices. They do not imply that averaging arbitrary nonconvex neural-network checkpoints always improves performance.^[4]

What convergence guarantees mean

There is no single "the convergence rate of SGD." A rate must name the problem class, oracle model, learning-rate sequence, output rule, and error criterion. Common criteria include objective suboptimality, squared distance to a minimizer, and squared gradient norm. They are not equivalent without further structure.^[2][3][4]

Scroll sideways for more →

The table summarizes representative results, not a promise for every implementation. Constants can depend on smoothness, strong convexity, initial distance, and noise variance. Different papers also place the square on different accuracy definitions. Ghadimi and Lan, for example, analyze a randomized output and give an oracle complexity for making the expected squared gradient norm small under a Lipschitz-gradient and bounded-variance model.^[3]

Convex objectives

For a convex objective, every local minimum is global. Strong convexity further provides a unique minimizer and relates objective error to distance from that minimizer. Those properties allow SGD analyses to turn expected descent into global optimization bounds.^[2][4]

Even in the strongly convex case, the result is conditional. With a fixed step size, persistent stochastic variance creates a residual term. With diminishing steps, exact asymptotic convergence may be possible, but a poor schedule can make the early iterations inefficient. Averaging can improve finite-time behavior or asymptotic efficiency under particular stochastic-approximation models.^[2][3][4]

For nonsmooth objectives, the current iterate can behave differently from an average. Shamir and Zhang showed that, under their assumptions, the final iterate has expected error $O(\log T/\sqrt{T})$ for general convex objectives and $O(\log T/T)$ for strongly convex objectives, while their polynomial-decay averaging obtains the corresponding optimal orders without those logarithmic factors.^[4]

Nonconvex objectives

For a differentiable nonconvex objective, $\nabla F(\theta)=0$ describes a stationary point. A stationary point may be a local minimum, a local maximum, or a saddle point. Consequently, a theorem that drives $\mathbb{E}\|\nabla F(\theta_R)\|^2$ below a tolerance does not identify which type was reached and does not prove global optimality.^[3]

Ghadimi and Lan analyze smooth objectives bounded below with an unbiased, bounded-variance stochastic first-order oracle. Their randomized stochastic-gradient method chooses an output from the generated iterates according to a specified probability distribution. In their accuracy convention, achieving $\mathbb{E}\|\nabla F(\theta_R)\|^2\leq\epsilon$ requires order $1/\epsilon^2$ stochastic-gradient calls in the noisy nonconvex setting. Altering the oracle, output rule, or assumptions changes the result.^[3]

Neural-network training adds features not captured by a basic smooth-oracle model, including nonsmooth activations, stateful layers, data augmentation, finite-precision arithmetic, and structured sampling. The basic theory remains useful for terminology and limiting cases, but an empirical training curve is not a direct verification of every theorem assumption.^[2]

Sampling schemes

With-replacement sampling

Independent uniform sampling with replacement is analytically convenient. For a finite sum, its component gradient is conditionally unbiased, and successive sample indices can be treated as independent. An example may be selected more than once before another example is selected at all.^[2][21]

Mini-batches may be sampled with or without replacement inside one update. For independent samples with equal covariance, the covariance of their mean falls in proportion to $1/b$, where $b$ is the batch size. Finite-population sampling without replacement has a correction factor. Either way, variance reduction per update does not automatically translate into an equal reduction in elapsed training time.^[2]

Random reshuffling

Random reshuffling draws a permutation of the $n$ examples and processes each one once before drawing another permutation for the next epoch. Conditional on the iterates already produced inside an epoch, the next component gradient is generally not an unbiased full-gradient estimate. The remaining indices depend on the earlier selections, and the parameter has changed since the epoch began.^[19][20][21]

The dependence is not merely a technical nuisance. It gives reshuffling behavior that can differ from independent SGD. For strongly convex finite sums with quadratic or additional smoothness structure, Gurbuzbalaban, Ozdaglar, and Parrilo proved rates under which reshuffling with averaging can outperform the usual with-replacement bound. Later analyses extended shuffling results to broader strongly convex and nonconvex finite-sum settings.^[19][20]

The improvement is not unconditional. Safran and Shamir constructed ill-conditioned quadratic problems for which worst-case gains from without-replacement sampling appear only after the number of epochs exceeds the condition number, up to their stated factors. Their result does not say reshuffling is useless in practice. It shows why "shuffling always converges faster" is too strong.^[21]

Nonuniform sampling

Uniform sampling is not the only way to preserve an unbiased gradient estimate. If index $i$ is sampled with probability $p_i>0$, the weighted estimator

$$g(\theta,i)=\frac{1}{n p_i}\nabla f_i(\theta)$$

is unbiased for the finite-sum gradient. The correction matters. Sampling difficult or high-norm examples more often without reweighting changes the objective being estimated.^[24]

Zhao and Zhang analyzed importance sampling for regularized stochastic optimization. Under their assumptions, choosing probabilities related to gradient norms or smoothness bounds can lower estimator variance and improve convergence bounds. Computing or maintaining useful probabilities has its own cost, so importance sampling is a method with a tradeoff, not free acceleration.^[24]

Gradient noise

For an unbiased estimator, stochastic gradient noise at $\theta_t$ can be written as

$$\zeta_t = g_t-\nabla F(\theta_t), \qquad \mathbb{E}[\zeta_t\mid\theta_t]=0.$$

Its covariance can depend on the current parameters, the data distribution, batch construction, and model. It need not be isotropic, independent over time, or Gaussian. Those simplified noise models can be useful approximations, but they should not be presented as defining properties of SGD.^[2][12][13]

The finite-batch noise also vanishes when the batch is the complete finite dataset and the computation is deterministic. For smaller batches, the noise scale generally changes with batch size. Correlated examples, augmentation, class-balanced sampling, and distributed data partitions can make the simple independent-sample $1/b$ variance rule inaccurate.^[2][15]

Research on the shape of SGD noise remains model-dependent. Gurbuzbalaban, Simsekli, and Zhu proved heavy-tailed stationary behavior for SGD in a quadratic linear-regression setting under specific multiplicative-noise conditions and reported supporting experiments beyond that setting.^[12] Battash, Wolf, and Lindenbaum later compared Gaussian and symmetric alpha-stable fits across selected discriminative and generative models and reported that the heavy-tailed model fit their measured noise better for most sampled parameters.^[13] These results do not establish that every SGD run has one universal alpha-stable noise law.

Noise is sometimes said to help SGD escape local minima. That phrase compresses several different questions: whether an iterate leaves a basin, which basin geometry is used, whether a stationary distribution exists, and whether the resulting predictor generalizes. Results for a diffusion model or a particular stochastic recurrence do not justify a universal claim that more noise improves optimization.^[12][13]

Generalization and implicit bias

Optimization error and generalization error are different. SGD can reduce a training objective while the held-out error improves, stays flat, or worsens. Early stopping, data order, batch size, and learning-rate schedules can act as algorithmic choices that change the selected solution, but their effects depend on the model and data.^[2][8][15]

Hardt, Recht, and Singer connected SGD to algorithmic stability. For Lipschitz and smooth losses, they bounded expected generalization error in terms of the learning-rate sequence, number of updates, sample size, and convexity assumptions. Their nonconvex result requires a decaying step-size bound and smoothness conditions. It is evidence for a specific stability mechanism, not proof that all long SGD runs generalize.^[8]

Another proposed explanation is that SGD prefers "flat" minima. Parameter-space flatness must be defined carefully. Dinh and colleagues showed that, for rectified networks, reparameterizations can change several common sharpness measures arbitrarily while leaving the represented function unchanged. A claim that one optimizer generalizes because its solution is flatter therefore needs a parameterization-aware measure and evidence connecting that measure to prediction.^[9]

Implicit-bias results can be exact in narrower models. Nacson, Srebro, and Soudry studied homogeneous linear classifiers on separable data with smooth monotone losses. Under their conditions, fixed-learning-rate SGD drives the predictor direction toward the maximum-margin direction, with a slow directional rate. This theorem is informative about a particular separable classification problem; it does not show that SGD selects a maximum-margin solution for every deep network.^[11]

SGD and adaptive methods

AdaGrad, RMSProp, and Adam rescale coordinates using accumulated gradient statistics. Plain SGD uses one global learning-rate scale, while momentum adds a state variable without the same coordinate-wise normalization. The algorithms can therefore follow different paths and select different solutions even when they reach similar training loss.^[10]

Wilson and colleagues constructed convex examples in which adaptive methods converge to solutions with worse test behavior than nonadaptive methods, and they reported experiments where tuned SGD or SGD with momentum generalized better than the tested adaptive methods. The paper is a counterexample to universal superiority of adaptive optimization, not a proof that SGD always generalizes better than AdaGrad, RMSProp, or Adam.^[10]

A fair comparison tunes each optimizer, holds the model and data pipeline fixed, states the compute budget, and reports both training and evaluation metrics. Equal learning-rate numbers are not comparable across algorithms because the update equations give those numbers different meanings.^[6][10][15]

Batch size and large-batch training

Batch size changes both the gradient estimator and the computation. Increasing it can reduce gradient variance and allow more data-parallel work per update. It can also reduce the number of parameter updates made per epoch. Whether it reduces time to a target depends on step efficiency, communication, tuning, and the point at which additional examples stop reducing the required number of steps.^[14][15][16]

Goyal and colleagues trained ResNet-50 on ImageNet with momentum SGD using a global mini-batch of 8,192 across 256 GPUs. With a linear learning-rate scaling rule, gradual warmup, and implementation-specific system work, they reported one-hour training, about 90 percent scaling efficiency from 8 to 256 GPUs, and validation error close to their batch-256 baseline. Their experiments also showed deterioration beyond the useful range, including worse results at larger batch sizes. The result is a bounded systems and optimization demonstration, not a general 8,192-example rule.^[14]

Shallue and colleagues studied 168,160 training runs across 35 workloads and recorded more than 71 million loss measurements. They found a common qualitative pattern: increasing batch size initially reduced the number of steps to a target, followed by diminishing returns and then saturation. The maximum useful batch varied widely. After tuning the learning rate, momentum, and schedule for each batch size, they found no evidence in those workloads that a larger batch necessarily degraded out-of-sample performance.^[15]

Those findings help separate two claims that are often mixed together. First, a large batch can stop reducing the number of required updates. Second, a large batch can change the best attainable evaluation error under a fixed protocol. The first was widespread in Shallue and colleagues' experiments. The second depended strongly on training budget and tuning, so an observed "generalization gap" can be confounded by comparing differently optimized runs.^[15]

McCandlish and colleagues proposed an empirical model in which a gradient-noise scale predicts a critical batch size and the tradeoff between the number of optimization steps and the number of examples processed. They tested the model across several domains. It is a useful workload-level diagnostic, but it is an empirical approximation with assumptions, not a fixed constant that can be transferred unchanged between models.^[16]

Distributed and parallel SGD

Synchronous data parallelism

In synchronous data parallelism, workers compute gradients on different examples at the same parameter vector, aggregate them, and apply one shared update. With exact averaging, this is mathematically a larger mini-batch update. Its systems cost includes communication and synchronization, and its optimization behavior depends on the resulting global batch and learning-rate configuration.^[14][15]

More workers do not guarantee proportional speedup. Communication, input processing, unequal worker speeds, and reduced step efficiency can dominate. Reporting only examples per second can hide a larger number of steps to the target, while reporting only steps can hide slower or more expensive steps.^[14][15]

Asynchronous shared-memory SGD

Asynchronous methods let workers read and update parameters without a global barrier. A worker may compute a gradient using stale values, and concurrent writes can interfere. HOGWILD! showed that lock-free asynchronous updates can converge and obtain near-linear multicore speedups under its sparse optimization model and stated assumptions. Dense neural-network training does not automatically satisfy that sparsity model.^[17]

Staleness is another source of error, not ordinary mini-batch noise. Its effect depends on delay, update sparsity, step size, and the architecture of the parameter server or shared memory. A convergence theorem for bounded delays or sparse coordinate conflicts should not be applied to an arbitrary asynchronous system without checking those conditions.^[17]

Local SGD

Local SGD allows each worker to perform several SGD steps on its own parameters before workers average their models. This reduces communication frequency but lets local copies drift apart. Stich proved linear worker speedup and reduced communication rounds for smooth strongly convex objectives under explicit bounds on the synchronization interval, variance, and learning-rate schedule. The paper did not claim the same theorem for every nonconvex deep network.^[18]

Local SGD is also related to federated optimization, but identical terminology does not imply identical assumptions. Non-identically distributed client data, partial participation, privacy mechanisms, and unreliable devices create problems beyond the homogeneous-worker setting analyzed in a basic local-SGD theorem.^[18]

Variance reduction and related finite-sum methods

Finite-sum methods can use the fact that an index refers to a component seen before. This information is absent in a pure expected-risk oracle with endlessly fresh samples. Variance-reduced methods combine new component gradients with stored or periodically refreshed information so that the estimator variance shrinks as optimization proceeds.^[2][22][23]

SVRG periodically computes a full gradient at a reference point and uses it to correct sampled component gradients. Johnson and Zhang proved linear convergence for smooth strongly convex finite-sum objectives under their stated conditions, without storing one full gradient vector per example.^[22]

SAGA stores component-gradient information and updates one entry at a time. Defazio, Bach, and Lacoste-Julien proved fast rates for smooth strongly convex finite sums and support for a composite proximal objective. The storage cost can be substantial, although some models admit compact representations.^[23]

These algorithms are not merely alternate brand names for plain SGD. Their update rules, memory or full-gradient costs, and assumptions differ. Their linear finite-sum convergence results do not imply linear convergence for an arbitrary nonconvex expected objective.^[2][22][23]

Random reshuffling and importance sampling also alter variance, but in different ways. Reshuffling introduces dependence by forbidding repeats within an epoch. Importance sampling changes selection probabilities and uses weights to preserve the desired expectation. Variance reduction, reshuffling, and nonuniform sampling should therefore be described separately even when they are combined in one system.^[19][20][24]

Practical evaluation

An SGD experiment is easier to interpret when the following items are fixed or reported:

1. Objective and reduction: State whether the loss is summed or averaged over a batch, how regularization enters, and whether any gradient clipping or projection changes the update.
2. Sampling rule: State whether sampling is with replacement, shuffled each epoch, fixed-order, class-balanced, weighted, or distributed across nonidentical shards.
3. Global and local batch: Report the number of examples contributing to one update and how they are divided among workers or accumulation steps.
4. Update equation: Give the precise momentum, dampening, weight-decay, and learning-rate conventions rather than only the optimizer name.
5. Schedule and budget: Report warmup, decay boundaries, stopping rule, number of updates, epochs, examples processed, and elapsed or accelerator time.
6. Selection and evaluation: Explain which checkpoint is evaluated and whether hyperparameters were selected on validation data separately for each optimizer or batch size.

These details matter because step count, example count, and wall-clock time can move in different directions. Shallue and colleagues found that tuning procedures and compute budgets explained many apparent disagreements about large-batch behavior. Bottou, Curtis, and Nocedal similarly emphasize that stochastic and batch methods trade per-step cost against asymptotic accuracy.^[2][15]

Diagnosing training behavior

A diverging or exploding training loss is consistent with an excessive effective learning rate, numerical instability, or a faulty gradient, but the curve alone does not identify which. Repeating a small deterministic batch, checking the loss reduction, inspecting gradient norms, and comparing one update to a reference implementation can isolate implementation errors before a large run.^[2]

A noisy training curve is not by itself a defect. Mini-batch losses are random measurements on changing batches, while a full-dataset or fixed-validation loss is a different statistic. Increasing the batch can smooth the estimate, but it also changes the optimization process and computational cost.^[2][15]

A plateau can reflect a learning rate that is too small, a fixed-step noise floor, ill conditioning, a saturated model, or a limit of the available data. A schedule change that lowers training loss does not establish better generalization. Both training and held-out metrics should be evaluated under a declared checkpoint-selection rule.^[2][8]

Common misconceptions

"Every stochastic gradient is unbiased"

Unbiasedness follows from a sampling and weighting rule. Uniform with-replacement sampling of finite-sum components gives an unbiased estimator. Random reshuffling, nonuniform sampling without correction, and stale asynchronous updates need separate treatment.^[2][19][24]

"A learning rate proportional to $1/\sqrt{t}$ meets the classical Robbins-Monro conditions"

It does not meet the square-summability condition. Its square is proportional to $1/t$, whose series diverges. The classical pair of conditions is satisfied by $c/t^p$ when $1/2<p\leq1$, subject to the other assumptions of the theorem.^[1][2]

"Constant-step SGD converges exactly"

With persistent stochastic variance, standard strongly convex results place constant-step SGD in a neighborhood whose size depends on the step and noise. Exact convergence requires additional structure, vanishing variance, averaging under a suitable result, or a diminishing step sequence.^[2]

"SGD avoids bad local minima because its noise is isotropic"

SGD noise can be anisotropic, state-dependent, correlated, and heavy-tailed in some settings. Nonconvex convergence results usually guarantee approximate stationarity, not selection of a favorable local minimum.^[3][12][13]

"Flatter parameters prove better generalization"

Many parameter-space sharpness measures change under reparameterizations that leave the network function unchanged. Any flatness explanation needs an invariant or otherwise justified measure and a demonstrated relation to prediction.^[9]

"Large batches necessarily generalize worse"

Some protocols have observed a gap, but it is not a universal law. In a large controlled study with per-batch tuning, Shallue and colleagues found no evidence that increasing batch size necessarily degraded out-of-sample performance across their 35 workloads. Large batches still showed diminishing optimization returns, and the useful range varied widely.^[15]

"SGD and Adam can be compared with the same learning rate"

The update equations assign different meanings to the learning-rate value. Optimizer comparisons need independent tuning and matched data, model, selection, and compute protocols.^[10][15]

See also

• Gradient Descent
• Mini-batch stochastic gradient descent
• Learning Rate
• Batch Size
• Momentum
• Backpropagation
• Convex Optimization
• Distributed training

References

1. ^Robbins, H., and Monro, S. (1951). "A Stochastic Approximation Method." The Annals of Mathematical Statistics, 22(3), 400-407. doi.org/...1177729586
2. ^Bottou, L., Curtis, F. E., and Nocedal, J. (2018). "Optimization Methods for Large-Scale Machine Learning." SIAM Review, 60(2), 223-311. doi.org/...16M1080173
3. ^Ghadimi, S., and Lan, G. (2013). "Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming." SIAM Journal on Optimization, 23(4), 2341-2368. doi.org/...120880811
4. ^Shamir, O., and Zhang, T. (2013). "Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes." Proceedings of the 30th International Conference on Machine Learning, 28(1), 71-79. proceedings.mlr.press/...shamir13
5. ^Polyak, B. T. (1964). "Some Methods of Speeding Up the Convergence of Iteration Methods." USSR Computational Mathematics and Mathematical Physics, 4(5), 1-17. doi.org/...0041-5553%2864%2990137-5
6. ^Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013). "On the Importance of Initialization and Momentum in Deep Learning." Proceedings of the 30th International Conference on Machine Learning, 28(3), 1139-1147. proceedings.mlr.press/...sutskever13
7. ^Loshchilov, I., and Hutter, F. (2017). "SGDR: Stochastic Gradient Descent with Warm Restarts." International Conference on Learning Representations. openreview.net/forum
8. ^Hardt, M., Recht, B., and Singer, Y. (2016). "Train Faster, Generalize Better: Stability of Stochastic Gradient Descent." Proceedings of the 33rd International Conference on Machine Learning, 48, 1225-1234. proceedings.mlr.press/...hardt16
9. ^Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. (2017). "Sharp Minima Can Generalize For Deep Nets." Proceedings of the 34th International Conference on Machine Learning, 70, 1019-1028. proceedings.mlr.press/...dinh17b
10. ^Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., and Recht, B. (2017). "The Marginal Value of Adaptive Gradient Methods in Machine Learning." Advances in Neural Information Processing Systems 30. papers.nips.cc/...504647f9d794f7d7b9bf341-Abstract
11. ^Nacson, M. S., Srebro, N., and Soudry, D. (2019). "Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate." Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, 89, 3051-3059. proceedings.mlr.press/...nacson19a
12. ^Gurbuzbalaban, M., Simsekli, U., and Zhu, L. (2021). "The Heavy-Tail Phenomenon in SGD." Proceedings of the 38th International Conference on Machine Learning, 139, 3964-3975. proceedings.mlr.press/...gurbuzbalaban21a
13. ^Battash, B., Wolf, L., and Lindenbaum, O. (2024). "Revisiting the Noise Model of Stochastic Gradient Descent." Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, 238, 4780-4788. proceedings.mlr.press/...battash24a
14. ^Goyal, P., Dollar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour." arXiv:1706.02677. arxiv.org/...1706.02677
15. ^Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G. E. (2019). "Measuring the Effects of Data Parallelism on Neural Network Training." Journal of Machine Learning Research, 20(112), 1-49. jmlr.org/...18-789
16. ^McCandlish, S., Kaplan, J., Amodei, D., and the OpenAI Dota Team. (2018). "An Empirical Model of Large-Batch Training." arXiv:1812.06162. arxiv.org/...1812.06162
17. ^Niu, F., Recht, B., Re, C., and Wright, S. J. (2011). "HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent." Advances in Neural Information Processing Systems 24. proceedings.neurips.cc/...5601cc6ddc1520e-Abstract
18. ^Stich, S. U. (2019). "Local SGD Converges Fast and Communicates Little." International Conference on Learning Representations. openreview.net/forum
19. ^Gurbuzbalaban, M., Ozdaglar, A., and Parrilo, P. A. (2021). "Why Random Reshuffling Beats Stochastic Gradient Descent." Mathematical Programming, 186, 49-84. doi.org/...s10107-019-01440-w
20. ^Nguyen, L. M., Tran-Dinh, Q., Phan, D. T., Nguyen, P. H., and van Dijk, M. (2021). "A Unified Convergence Analysis for Shuffling-Type Gradient Methods." Journal of Machine Learning Research, 22(207), 1-44. jmlr.org/...20-1238
21. ^Safran, I., and Shamir, O. (2021). "Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned Problems." Advances in Neural Information Processing Systems 34. proceedings.neurips.cc/...8fc4cdb3065e8ce-Abstract
22. ^Johnson, R., and Zhang, T. (2013). "Accelerating Stochastic Gradient Descent Using Predictive Variance Reduction." Advances in Neural Information Processing Systems 26. papers.nips.cc/...bcc5e5d1c6e28598e8cbbe8-Abstract
23. ^Defazio, A., Bach, F., and Lacoste-Julien, S. (2014). "SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives." Advances in Neural Information Processing Systems 27. papers.nips.cc/...d6fb3a55cd7cc578165f058-Abstract
24. ^Zhao, P., and Zhang, T. (2015). "Stochastic Optimization with Importance Sampling for Regularized Loss Minimization." Proceedings of the 32nd International Conference on Machine Learning, 37, 1-9. proceedings.mlr.press/...zhaoa15