# Backpropagation

> Source: https://aiwiki.ai/wiki/backpropagation
> Updated: 2026-07-29
> Fact-checked: 2026-07-29
> Categories: Deep Learning, Machine Learning, Neural Networks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Backpropagation." aiwiki.ai, 29 Jul 2026. https://aiwiki.ai/wiki/backpropagation
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Backpropagation is an algorithm for computing derivatives of a scalar objective with respect to variables in a layered or otherwise composed computation. It applies the chain rule from the output of a [computational graph](https://aiwiki.ai/wiki/computational_graph) toward its inputs, accumulating the contribution from every downstream path. In neural-network training, those derivatives are usually gradients with respect to weights and biases. Backpropagation computes the gradients; a separate [optimizer](https://aiwiki.ai/wiki/optimizer) decides how, or whether, to update the parameters.

The method is the neural-network specialization of reverse-mode [automatic differentiation](https://aiwiki.ai/wiki/automatic_differentiation). It is especially useful in practice when a computation has many inputs or parameters and a scalar output such as a [loss function](https://aiwiki.ai/wiki/loss_function), because one reverse sweep can compute derivatives of that scalar with respect to all participating inputs.[1][2] Modern software applies the same principle to computation graphs containing tensor operations, branches, shared parameters, and repeated subcomputations.

Backpropagation does not by itself specify a network architecture, objective, data set, regularization method, learning rate, or optimization rule. It can be used with [gradient descent](https://aiwiki.ai/wiki/gradient_descent), but the two terms are not synonyms. It also does not claim that biological nervous systems use the same mechanism.

## Scope and terminology

Several related terms are often conflated:

| Term | What it denotes |
| --- | --- |
| Backpropagation | Reverse accumulation of derivatives through a composed computation, conventionally discussed for neural networks |
| Reverse-mode automatic differentiation | The more general program-transformation or graph-execution technique that evaluates vector-Jacobian products from outputs toward inputs |
| Gradient descent | An optimization rule that changes variables in the negative-gradient direction |
| Training step | A broader procedure that can include a forward evaluation, loss construction, backpropagation, gradient processing, an optimizer update, and state updates |
| Numerical differentiation | Approximation of derivatives from perturbed function values, often by finite differences |
| Symbolic differentiation | Algebraic construction and simplification of derivative expressions |

Automatic differentiation evaluates derivatives by composing derivative rules for the elementary operations that actually ran. It is neither finite-difference approximation nor symbolic manipulation of a complete closed-form expression. Its computed values are subject to ordinary finite-precision rounding, and its validity depends on the derivative rules assigned to the graph's operations.[2]

The word "backward" describes the dependency order of derivative accumulation, not a reversal of the model's numerical forward computation. A reverse pass uses values or metadata from the forward pass and propagates sensitivity information from an output toward its ancestors.

## Computation-graph formulation

Consider a directed acyclic graph whose nodes are values. Each non-input value is produced by a local operation from its parents. Let the scalar objective be `L`, and define the adjoint of a node `v_i` as its total derivative:

$$
\bar{v}_i = \frac{\partial L}{\partial v_i}.
$$

The reverse sweep starts with:

$$
\bar{L} = \frac{\partial L}{\partial L} = 1.
$$

If `children(i)` is the set of nodes that directly use `v_i`, then the chain rule gives:

$$
\bar{v}_i
=
\sum_{j \in \operatorname{children}(i)}
\left(\frac{\partial v_j}{\partial v_i}\right)^{\mathsf T}
\bar{v}_j.
$$

This equation contains two essential features of backpropagation:

- Locality: each operation needs a rule for applying the transpose of its local derivative to an incoming adjoint.
- Accumulation: if a value influences the objective along several paths, the contributions from all of those paths are added.

A reverse execution therefore proceeds in a reverse topological order:

1. Evaluate the forward graph and retain, or arrange to reconstruct, the values required by local derivative rules.
2. Seed the chosen output with an adjoint, normally `1` for a scalar loss.
3. Visit operations in reverse dependency order.
4. Apply each operation's local vector-Jacobian product.
5. Add contributions at fan-out points and shared variables.
6. Return or accumulate adjoints for the requested inputs and parameters.

The addition at a branch is not optional. For example, if a scalar `x` is used in both `u = x^2` and `v = sin(x)`, and `L = uv`, then:

$$
\frac{dL}{dx}
=
\frac{\partial L}{\partial u}\frac{du}{dx}
+
\frac{\partial L}{\partial v}\frac{dv}{dx}
=
2x\sin(x) + x^2\cos(x).
$$

An implementation that overwrites one contribution instead of adding both produces an incorrect derivative. The same issue appears when a parameter is reused at multiple layers, time steps, or branches.

### Local rules and tensor shapes

Backpropagation normally applies local derivative products without constructing a dense Jacobian. Common rules illustrate how this works. If `y = x_1 + x_2`, both inputs receive the incoming adjoint:

$$
\bar{x}_1 \mathrel{+}= \bar{y},
\qquad
\bar{x}_2 \mathrel{+}= \bar{y}.
$$

For an elementwise product `y = x_1` elementwise-multiplied by `x_2`:

$$
\bar{x}_1 \mathrel{+}= \bar{y}\odot x_2,
\qquad
\bar{x}_2 \mathrel{+}= \bar{y}\odot x_1.
$$

For matrix multiplication:

$$
Y = AB,
$$

the corresponding reverse products are:

$$
\bar{A} \mathrel{+}= \bar{Y}B^{\mathsf T},
\qquad
\bar{B} \mathrel{+}= A^{\mathsf T}\bar{Y}.
$$

These formulas use the forward operands but never form the full Jacobian of matrix multiplication. This is a major practical advantage of operation-specific vector-Jacobian products.

Shape transformations have derivative rules too. A reshape maps the incoming adjoint back to the original shape. A transpose applies the inverse permutation. A slice or gather scatters contributions into the selected source positions, adding them when indices repeat. A sum reduction broadcasts its incoming adjoint over the reduced positions. A mean reduction does the same and divides by the number of reduced elements.

Broadcasting requires the reverse operation to sum over broadcast axes. If a bias vector is added to every row of a batch, its gradient is the sum of the row-wise adjoints, not one selected row. Incorrect handling of broadcasting can produce a result with a plausible value but the wrong shape or scale.

Maxima, sorting, indexing by discrete choices, and repeated indices need more care. A maximum is nondifferentiable at a tie, so its derivative depends on a selected convention. Discrete indices ordinarily have no useful gradient, while the selected floating-point values can still receive adjoints. These are properties of the local operation, not exceptions to the graph-level accumulation rule.

## Layered-network derivation

For a feedforward network, use column vectors and let layer `l` have weight matrix `W^(l)`, bias vector `b^(l)`, pre-activation `z^(l)`, activation `a^(l)`, and elementwise activation function `phi^(l)`. The forward equations are:

$$
a^{(0)} = x,
$$

$$
z^{(l)} = W^{(l)}a^{(l-1)} + b^{(l)},
$$

$$
a^{(l)} = \phi^{(l)}\left(z^{(l)}\right).
$$

Let the final scalar objective be:

$$
L = \ell\left(a^{(m)}, y\right).
$$

Define the layer error signal as the derivative with respect to the pre-activation:

$$
\delta^{(l)} = \nabla_{z^{(l)}} L.
$$

For a general final activation, the output-layer signal is:

$$
\delta^{(m)}
=
J_{\phi^{(m)}}\left(z^{(m)}\right)^{\mathsf T}
\nabla_{a^{(m)}}\ell.
$$

For an elementwise hidden activation, the reverse recurrence is:

$$
\delta^{(l)}
=
\left(\left(W^{(l+1)}\right)^{\mathsf T}
\delta^{(l+1)}\right)
\odot
\phi^{(l)\prime}\left(z^{(l)}\right),
$$

where `odot` denotes elementwise multiplication. The parameter derivatives are:

$$
\nabla_{W^{(l)}}L
=
\delta^{(l)}
\left(a^{(l-1)}\right)^{\mathsf T},
$$

$$
\nabla_{b^{(l)}}L
=
\delta^{(l)}.
$$

These shapes are part of the derivation. If `W^(l)` has shape `n_l` by `n_(l-1)`, then `delta^(l)` has length `n_l`, and its outer product with `a^(l-1)` has exactly the shape of the weight matrix.

For a batch of `B` examples stored as columns, the bias gradient sums the per-example signals. If the reported loss is the batch mean, the factor `1/B` belongs in the gradients:

$$
\nabla_{W^{(l)}}L_{\mathrm{mean}}
=
\frac{1}{B}
\Delta^{(l)}
\left(A^{(l-1)}\right)^{\mathsf T},
$$

$$
\nabla_{b^{(l)}}L_{\mathrm{mean}}
=
\frac{1}{B}
\Delta^{(l)}\mathbf{1}.
$$

Changing a loss from a sum to a mean changes the gradient scale. This is a common source of apparent disagreements between a hand derivation and a framework result.

Libraries may implement a loss and final activation as one fused operation. For example, a fused log-softmax and cross-entropy derivative can be evaluated without explicitly materializing a full activation Jacobian. Such a simplification is an operation-specific vector-Jacobian product, not a change to the chain rule.

## Vector-Jacobian products

Suppose a function maps `n` inputs to `m` outputs:

$$
f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}.
$$

Its Jacobian at `x` is:

$$
J_f(x) \in \mathbb{R}^{m \times n}.
$$

Forward-mode automatic differentiation propagates a tangent vector `r` and evaluates a Jacobian-vector product:

$$
J_f(x)r.
$$

Reverse mode propagates an output cotangent `s` and evaluates a vector-Jacobian product, equivalently a transposed-Jacobian-vector product:

$$
J_f(x)^{\mathsf T}s.
$$

For a scalar objective, `m = 1` and the seed is normally `s = 1`, so a reverse sweep returns the full input gradient. For a non-scalar output, a backward call needs an explicit seed or an implicit reduction to a scalar. One reverse sweep then computes the derivative of the selected linear combination of outputs, not the entire Jacobian.

| Mode | Primitive product | A favorable dimensional regime | Full Jacobian construction |
| --- | --- | --- | --- |
| Forward mode | `Jr` | Few differentiated inputs and many outputs | Repeat over input basis directions |
| Reverse mode | `J^T s` | Many differentiated inputs and few outputs | Repeat over output basis directions |

This comparison is dimensional, not an unconditional performance promise. Operation mix, batching, compiler transformations, sparsity, memory traffic, and hardware can change measured performance. JAX's official documentation exposes the distinction directly through Jacobian-vector products and vector-Jacobian products.[3]

## Computational cost and storage

Reverse accumulation avoids evaluating one complete derivative program per parameter. In the arithmetic-operation model used by automatic-differentiation analyses, the gradient of a scalar function can be obtained at a small constant multiple of the cost of evaluating the function. A widely cited survey reports a bound below six for its model and a typical factor around two to three, while also treating storage as a separate cost.[2] These are operation-count statements, not guarantees about wall-clock time on every device or software stack.

The reverse pass needs enough forward information to evaluate each local derivative. Depending on the operation, that information might be an input, an output, a shape, an index selection, a mask, or other metadata. A framework may:

- retain the required value;
- recompute it later;
- save a compressed representation;
- offload it to another memory tier; or
- use a fused derivative rule that avoids materializing it.

Consequently, "backpropagation stores every activation" is too broad. Some values must remain available or be reconstructed, but the required set is determined by the operations and the implementation. PyTorch, for example, documents that its autograd graph saves selected tensors needed by backward formulas and releases them according to graph lifetime rules.[4]

[Gradient checkpointing](https://aiwiki.ai/wiki/gradient_checkpointing), also called activation checkpointing or rematerialization, trades additional computation for lower saved-activation memory. For a chain-like network of `n` layers, Chen and colleagues described a schedule with order `sqrt(n)` feature-map memory and roughly one additional forward pass. They also analyzed a more extreme order `log(n)` memory schedule with order `n log(n)` forward recomputation.[5] Those results concern particular graph structures and schedules; they are not a universal bound for every dynamic graph.

Parameter storage, parameter gradients, optimizer state, communication buffers, and temporary kernels are separate from saved activations. Reducing activation memory does not necessarily reduce all of those components.

## Behavior in major frameworks

The interfaces differ, but common systems implement the same derivative products:

| System | Relevant interface | Documented behavior |
| --- | --- | --- |
| [PyTorch](https://aiwiki.ai/wiki/pytorch) | `Tensor.backward`, `torch.autograd.grad`, `torch.func` | Autograd records a graph of executed tensor operations, uses reverse automatic differentiation, and saves operation-specific tensors as needed. The eager graph is rebuilt on each iteration.[4] |
| [TensorFlow](https://aiwiki.ai/wiki/tensorflow) | `tf.GradientTape` | A tape records eligible operations involving watched values and later computes derivatives with respect to requested sources. Persistent tapes permit more than one gradient query but retain resources longer.[6] |
| [JAX](https://aiwiki.ai/wiki/jax) | `grad`, `vjp`, `jvp`, and compositions of transformations | `grad` is built on reverse mode; `vjp` exposes reverse products and `jvp` exposes forward products.[3] |

Framework details matter:

- A tensor must participate in the recorded computation and be marked or watched according to the framework's rules.
- An intentional `detach`, stop-gradient operation, conversion to a non-differentiable representation, or state mutation can cut a dependency.
- A custom primitive needs a correct derivative rule for every differentiation mode the application uses.
- Backward calls may accumulate into existing gradient buffers rather than replacing them.
- Retaining a derivative graph for repeated or higher-order differentiation changes its lifetime and memory use.

Dynamic control flow is differentiated along the operations that execute. That does not mean every branch of the source program contributes to a particular derivative evaluation. It also does not imply that all compilers represent or optimize the graph in the same way.

### Derivative scope and objective construction

A derivative is always taken with respect to specified sources. Values treated as constants do not receive gradients even if they are numerically identical to a differentiable value elsewhere. A leaf parameter, an intermediate activation, and a copied or detached tensor can therefore have different derivative behavior.

If an objective is a weighted sum:

$$
L = \sum_{k=1}^{K}\lambda_k L_k,
$$

linearity gives:

$$
\nabla_{\theta}L
=
\sum_{k=1}^{K}\lambda_k\nabla_{\theta}L_k.
$$

The coefficients, reductions, masks, and regularization terms are part of the mathematical objective. Backpropagation cannot infer an intended objective that was not encoded in the graph. If a regularization penalty is added outside the recorded computation, its derivative is absent. If a mask is normalized by the number of valid examples rather than the batch size, that denominator changes the gradient.

Microbatch accumulation and distributed reduction add another scale choice. Summing gradients across `M` equal-sized microbatches gives the gradient of the summed microbatch losses. Dividing by `M` gives the gradient of their mean, provided each microbatch loss uses compatible reductions. A distributed all-reduce can sum or average across workers. The resulting scale should be established from the exact framework and training code, not guessed from the word "average."

Randomness also belongs to the forward computation. Backpropagation through a sampled continuous variable requires a differentiable path, often supplied by a reparameterization. A discrete sampling decision does not acquire a derivative merely because later operations are differentiable. Gradient estimators for such choices are additional methods, not automatic consequences of ordinary backpropagation.[2]

## Differentiability and numerical behavior

Backpropagation composes local derivatives. If a primitive is differentiable at the evaluated point and its derivative rule is correct, the chain rule applies. Practical graphs also contain operations with nondifferentiable points, discontinuities, integer-valued outputs, or undefined values.

At a nondifferentiable point, there may be no unique mathematical gradient. A framework can choose a subgradient or another documented convention. PyTorch, for example, documents a sequence of conventions for operations such as ReLU or square root at zero.[4] A returned value at such a point should not be mistaken for proof that the classical derivative exists.

Invalid forward values can contaminate a reverse pass. Masking an infinity or `NaN` after an invalid operation does not necessarily remove that operation from the recorded graph. PyTorch's documentation gives division by zero as an example in which a masked forward result can still produce a `NaN` gradient.[4]

Finite precision creates additional hazards:

- products of many local derivatives can underflow, overflow, or lose significant digits;
- mixed-precision execution can require loss scaling or other range management;
- reductions can produce different rounding under different execution orders;
- numerically equivalent forward expressions can have different derivative stability;
- derivative rules for complex variables require an explicit convention.

These issues affect the computed number without changing the underlying chain-rule identity.

## What a correct gradient does and does not imply

For a differentiable scalar objective, a computed gradient is a local first-order sensitivity. Its component for parameter `theta_i` reports the instantaneous rate of change with all other coordinates held fixed:

$$
\left(\nabla_{\theta}L\right)_i
=
\frac{\partial L}{\partial \theta_i}.
$$

For a sufficiently small perturbation `Delta theta`, the first-order approximation is:

$$
L(\theta+\Delta\theta)
\approx
L(\theta)
+
\nabla_{\theta}L^{\mathsf T}\Delta\theta.
$$

The omitted higher-order terms matter for finite steps. A negative-gradient direction is a local descent direction when the gradient is nonzero and the objective is differentiable, but an arbitrary step length along that direction can still increase the objective. Choosing and adapting the step is an optimizer problem.

A correct gradient does not guarantee:

- that the objective is convex;
- that an optimizer will reach a global or useful minimum;
- that training data and evaluation data have the same distribution;
- that lower training loss improves generalization;
- that the objective measures the desired behavior;
- that a finite-precision training run is stable; or
- that the model is identifiable from the available data.

Conversely, a failed training run does not by itself show that backpropagation was implemented incorrectly. The cause can lie in optimization, initialization, data, the objective, numerical range, model capacity, or gradient computation. Separating those layers of diagnosis prevents the term "backpropagation failure" from absorbing unrelated problems.

## Verifying an implementation

A gradient check compares an analytical or automatically differentiated result with a derivative estimated from perturbed function values. For a scalar function and coordinate direction `e_i`, a central finite difference is:

$$
\frac{\partial f}{\partial x_i}
\approx
\frac{f(x+\varepsilon e_i)-f(x-\varepsilon e_i)}
{2\varepsilon}.
$$

The truncation error decreases as `epsilon` becomes smaller, but floating-point cancellation eventually increases the numerical error. A useful check therefore uses a scale-appropriate step, higher precision when possible, and inputs away from known nondifferentiable points. PyTorch's `gradcheck` documentation describes comparisons between analytical automatic-differentiation results and finite-difference estimates, including separate treatment of real and complex functions.[7]

Directional checks can be cheaper for high-dimensional inputs. For a direction `r`, compare:

$$
\nabla f(x)^{\mathsf T}r
$$

with:

$$
\frac{f(x+\varepsilon r)-f(x-\varepsilon r)}
{2\varepsilon}.
$$

A sound test suite also checks:

- expected tensor shapes and reduction factors;
- fan-out and shared-parameter accumulation;
- zero-length, masked, or degenerate inputs;
- custom operations against a trusted reference;
- first and higher derivatives separately;
- finite outputs and gradients under representative ranges;
- behavior at intentionally nondifferentiable points;
- gradient-buffer clearing or accumulation semantics.

A finite-difference match is strong diagnostic evidence but not a proof for all inputs. Two implementations can also agree while sharing the same modeling error, such as reducing the wrong objective.

## Backpropagation through time

Backpropagation through time, or BPTT, applies reverse-mode differentiation to a recurrent computation unrolled across time. Let a recurrent state satisfy:

$$
h_t = \Phi(h_{t-1}, x_t; \theta),
$$

and let the sequence objective be:

$$
L = \sum_{t=1}^{T}\ell_t(h_t).
$$

Define the state adjoint:

$$
\bar{h}_t = \nabla_{h_t}L.
$$

With a terminal condition of zero beyond the sequence, the reverse recurrence is:

$$
\bar{h}_t
=
\nabla_{h_t}\ell_t
+
\left(\frac{\partial h_{t+1}}{\partial h_t}\right)^{\mathsf T}
\bar{h}_{t+1}.
$$

Because the same parameter `theta` is reused, its contributions from all time steps must be added:

$$
\nabla_{\theta}L
=
\sum_{t=1}^{T}
\left(\frac{\partial h_t}{\partial \theta}\right)_{\mathrm{local}}^{\mathsf T}
\bar{h}_t.
$$

The local partial derivative holds the incoming state `h_(t-1)` fixed. If a per-step loss depends directly on `theta`, its direct partial derivative must also be added. This distinction prevents double-counting dependencies that already flow through the state adjoints.

Werbos's 1990 paper gave a systematic account of BPTT for dynamic and recurrent systems.[8] The method is ordinary backpropagation on the time-unrolled graph, not a separate chain rule.

Truncated BPTT limits how far state adjoints propagate. An implementation commonly detaches the recurrent state at window boundaries or otherwise stops the reverse traversal. Unless dependencies beyond the window make no contribution, the resulting gradient differs from the full-sequence gradient. Truncation can reduce memory and latency, but it changes the derivative being computed.

This terminology should not be applied indiscriminately to transformers. A conventional transformer processes a finite computation graph and is differentiated through its layers and token interactions. It does not become truncated BPTT merely because its input has a finite context window. BPTT specifically concerns reverse differentiation through recurrent state transitions over time.

## Vanishing and exploding gradients

In a deep or time-unrolled graph, a distant contribution contains a product of local Jacobians. For recurrent states:

$$
\frac{\partial h_t}{\partial h_k}
=
\frac{\partial h_t}{\partial h_{t-1}}
\frac{\partial h_{t-1}}{\partial h_{t-2}}
\cdots
\frac{\partial h_{k+1}}{\partial h_k},
$$

with the product ordered according to the chain rule. Repeated multiplication can shrink components toward zero or enlarge them sharply. The result depends on singular directions, activation derivatives, inputs, states, and how those directions align across steps. A single rule such as "a spectral norm above one means gradients explode" is not valid for every nonlinear, time-varying network.

Bengio, Simard, and Frasconi showed that learning long-term dependencies with gradient-based recurrent training becomes increasingly difficult as the dependency duration grows.[9] Pascanu, Mikolov, and Bengio later analyzed vanishing and exploding behavior using products of recurrent Jacobians and distinguished sufficient from necessary conditions in specific model settings.[10]

Gradient norm clipping replaces a raw gradient `g` by:

$$
g_{\mathrm{clip}}
=
g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right),
$$

where `tau` is a positive threshold. Pascanu and colleagues proposed and experimentally evaluated this form as a response to exploding gradients in recurrent-network training.[10] It bounds the norm of the gradient passed to the optimizer. It does not necessarily bound the eventual parameter change when the optimizer also uses momentum, adaptive preconditioning, weight decay, or other state.

The [Long Short-Term Memory (LSTM)](https://aiwiki.ai/wiki/lstm) architecture was introduced in part to address decaying error flow over long intervals.[11] LSTM does not make every gradient well-conditioned, and clipping does not solve vanishing gradients. Architecture, initialization, normalization, residual paths, objective design, sequence length, and numerical precision can all affect gradient propagation.

### Monitoring gradient propagation

Per-layer or per-time-step gradient norms can reveal where sensitivities shrink, grow, or become non-finite. The interpretation must account for parameter count and tensor shape: a large layer can have a larger total norm even when its typical component is not unusually large. Useful diagnostics can include the global norm, layer-wise norms, selected component statistics, activation ranges, update-to-parameter ratios, and the first operation that produces a non-finite value.

Clipping can keep a run numerically usable while concealing the source of repeated explosions. A report should therefore record how often clipping activates and the unclipped norm when feasible. Similarly, loss scaling can prevent underflow without repairing an incorrectly disconnected graph.

Vanishing gradients are also not equivalent to inactive parameters in every sense. Saturated activations can have small local derivatives even when a different parameter setting would materially change the model. Symmetries can create flat directions. A zero first derivative can occur at a minimum, maximum, saddle point, or nondifferentiable convention. Gradient statistics need context from the forward state and objective.

## Higher-order derivatives

Backpropagation produces first-order vector-Jacobian products, but automatic-differentiation transformations can be composed. Examples include:

- forward mode applied to a reverse-mode gradient;
- reverse mode applied to a forward-mode product;
- reverse mode applied again when the derivative graph is itself differentiable.

For a scalar function with Hessian `H`, many algorithms need `Hr` for a vector `r`, not the full Hessian. Pearlmutter derived an exact procedure for Hessian-vector products by differentiating the gradient computation in a direction, with computational cost comparable to a gradient evaluation in the analyzed setting.[12] Modern systems can express such products through compositions of JVP and VJP operations.[3]

Higher-order differentiation requires suitable derivative rules for the operations in the first derivative graph. It can also retain more graph state, increase memory use, and expose nondifferentiability that was not relevant to a first derivative.

## Biological interpretation and alternatives

Backpropagation is a computational method, not an established description of cortical learning. Strict implementations pose biological questions about how a system could transport appropriately signed error signals, use feedback related to forward synaptic weights, separate forward activity from error information, and coordinate updates across layers. A 2020 review concluded that these issues remain active research questions while surveying mechanisms that approximate some core computations.[13]

Feedback alignment replaces exact transposed forward weights in the hidden-layer error path with fixed random feedback weights. Lillicrap and colleagues showed comparable behavior to backpropagation in particular linear, nonlinear, handwritten-digit, and function-approximation experiments.[14] Those experiments demonstrate that exact weight symmetry was unnecessary in the tested systems. They do not establish that feedback alignment matches backpropagation on arbitrary architectures or that a biological nervous system uses it.

The forward-forward algorithm replaces a forward and backward pair with positive and negative forward passes and local layer objectives. Its original report explicitly described the work as preliminary and demonstrated it on a few small problems.[15] It is therefore best treated as a research proposal, not as a general replacement with established parity.

Other work studies local losses, target propagation, equilibrium-based learning, perturbation methods, and learned feedback pathways. These approaches answer different constraints and should be compared on specified models, data, computation budgets, and accuracy criteria rather than grouped as proven substitutes.

## History

The history of backpropagation is distributed across numerical analysis, control, dynamic systems, and neural-network research. A historical survey by Andreas Griewank documents multiple independent incarnations of reverse differentiation from the 1960s onward and discusses Seppo Linnainmaa's work on reversing computational expressions in 1970 and 1976.[16] This record does not support assigning the general reverse-mode idea to a single neural-network paper.

Werbos applied ordered derivatives to learning and dynamic-system problems and later presented BPTT in a 1990 *Proceedings of the IEEE* article.[8] Rumelhart, Hinton, and Williams circulated a 1985 report on the generalized delta rule, then published a concise account in *Nature* in 1986.[1][18] Their longer chapter also acknowledged independent related derivations by David Parker and Yann LeCun.[18] The 1986 work helped establish backpropagation as a practical method for learning internal representations in multilayer networks.

LeCun and colleagues subsequently demonstrated backpropagation in a network for handwritten ZIP-code recognition in 1989.[17] These works belong to a longer development rather than a clean sequence in which one publication contains every part of the modern method.

## Common misconceptions

| Misconception | Correction |
| --- | --- |
| Backpropagation updates the weights | It computes derivatives. An optimizer applies an update rule. |
| Backpropagation and gradient descent are the same | Backpropagation supplies gradients; gradient descent is one way to use them. |
| A reverse pass always returns a full Jacobian | A seeded reverse pass returns a vector-Jacobian product. A scalar loss with seed `1` yields a gradient. |
| Every forward tensor must be stored | Only values required by derivative rules must remain available, and they can sometimes be recomputed, compressed, fused, or offloaded. |
| Automatic differentiation has no numerical error | It avoids finite-difference truncation but still uses finite-precision arithmetic. |
| A framework's value at a kink is the unique derivative | A classical derivative may not exist there; the framework uses a convention. |
| A zero gradient proves a parameter is globally irrelevant | It only reports local first-order sensitivity at the evaluated point and along the recorded graph. |
| Gradient clipping caps the parameter update | It caps the selected gradient norm before later optimizer transformations. |
| A transformer's finite context window is truncated BPTT | Truncated BPTT stops reverse propagation across recurrent state transitions; a finite nonrecurrent graph is a different case. |
| Backpropagation is known to be the brain's learning algorithm | Its biological interpretation remains unsettled. |

## Practical reporting checklist

A reproducible description of a backpropagation-based experiment should identify:

- the exact scalar objective and whether each reduction is a sum or mean;
- the differentiated parameters and any frozen or detached paths;
- the framework and relevant derivative or custom-operation rules;
- numerical precision and any loss scaling;
- accumulation across microbatches, examples, devices, or time steps;
- clipping, normalization, or other processing applied before the optimizer;
- whether the graph is retained or rematerialized;
- the BPTT window and state-detachment policy for recurrent models;
- the method used to check custom gradients;
- how non-finite values and nondifferentiable points are handled.

This separation makes it possible to determine whether a reported failure comes from the derivative computation, the objective, the optimizer, numerical range, or the surrounding training procedure.

## See also

- [Activation Function](https://aiwiki.ai/wiki/activation_function)
- [Deep Learning](https://aiwiki.ai/wiki/deep_learning)
- [Deep Neural Network](https://aiwiki.ai/wiki/deep_neural_network)
- [Machine Learning](https://aiwiki.ai/wiki/machine_learning)
- [Perceptron](https://aiwiki.ai/wiki/perceptron)
- [Rectified Linear Unit (ReLU)](https://aiwiki.ai/wiki/rectified_linear_unit_relu)
- [Geoffrey Hinton](https://aiwiki.ai/wiki/geoffrey_hinton)
- [Yann LeCun](https://aiwiki.ai/wiki/yann_lecun)

## References

1. Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning representations by back-propagating errors." *Nature* 323, 533-536 (1986). https://www.nature.com/articles/323533a0
2. Baydin, Atilim Gunes, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. "Automatic Differentiation in Machine Learning: a Survey." *Journal of Machine Learning Research* 18(153), 1-43 (2018). https://www.jmlr.org/papers/v18/17-468.html
3. JAX documentation. "Forward- and reverse-mode autodiff in JAX." Accessed July 28, 2026. https://docs.jax.dev/en/latest/jacobian-vector-products.html
4. PyTorch documentation. "Autograd mechanics." Accessed July 28, 2026. https://docs.pytorch.org/docs/stable/notes/autograd.html
5. Chen, Tianqi, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. "Training Deep Nets with Sublinear Memory Cost." arXiv:1604.06174 (2016). https://arxiv.org/abs/1604.06174
6. TensorFlow Core documentation. "Introduction to gradients and automatic differentiation." Accessed July 28, 2026. https://www.tensorflow.org/guide/autodiff
7. PyTorch documentation. "Gradcheck mechanics." Accessed July 28, 2026. https://docs.pytorch.org/docs/stable/notes/gradcheck.html
8. Werbos, Paul J. "Backpropagation Through Time: What It Does and How to Do It." *Proceedings of the IEEE* 78(10), 1550-1560 (1990). https://doi.org/10.1109/5.58337
9. Bengio, Yoshua, Patrice Simard, and Paolo Frasconi. "Learning Long-Term Dependencies with Gradient Descent Is Difficult." *IEEE Transactions on Neural Networks* 5(2), 157-166 (1994). https://pubmed.ncbi.nlm.nih.gov/18267787/
10. Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." *Proceedings of the 30th International Conference on Machine Learning*, PMLR 28(3), 1310-1318 (2013). https://proceedings.mlr.press/v28/pascanu13.html
11. Hochreiter, Sepp, and Jurgen Schmidhuber. "Long Short-Term Memory." *Neural Computation* 9(8), 1735-1780 (1997). https://direct.mit.edu/neco/article/9/8/1735/6109/Long-Short-Term-Memory
12. Pearlmutter, Barak A. "Fast Exact Multiplication by the Hessian." *Neural Computation* 6(1), 147-160 (1994). https://doi.org/10.1162/neco.1994.6.1.147
13. Lillicrap, Timothy P., Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. "Backpropagation and the brain." *Nature Reviews Neuroscience* 21, 335-346 (2020). https://www.nature.com/articles/s41583-020-0277-3
14. Lillicrap, Timothy P., Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. "Random synaptic feedback weights support error backpropagation for deep learning." *Nature Communications* 7, 13276 (2016). https://www.nature.com/articles/ncomms13276
15. Hinton, Geoffrey. "The Forward-Forward Algorithm: Some Preliminary Investigations." arXiv:2212.13345 (2022). https://arxiv.org/abs/2212.13345
16. Griewank, Andreas. "Who Invented the Reverse Mode of Differentiation?" *Documenta Mathematica*, Extra Volume ISMP, 389-400 (2012). https://ems.press/books/dms/251/4949
17. LeCun, Yann, Bernhard Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne Hubbard, and Lawrence D. Jackel. "Backpropagation Applied to Handwritten Zip Code Recognition." *Neural Computation* 1(4), 541-551 (1989). https://doi.org/10.1162/neco.1989.1.4.541
18. Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning Internal Representations by Error Propagation." Institute for Cognitive Science Report 8506 (1985); subsequently published as a chapter in *Parallel Distributed Processing*, volume 1 (1986). https://www.cs.toronto.edu/~bonner/courses/2016s/csc321/readings/Learning%20representations%20by%20back-propagating%20errors.pdf