Partial derivative

See also: Machine learning terms

A partial derivative measures how a multivariable function changes when one of its inputs is varied while every other input is held fixed. For a function $f(x_1, x_2, \ldots, x_n)$, the partial derivative with respect to the i-th variable is written as $\frac{\partial f}{\partial x_i}$. Partial derivatives are the building block of multivariable calculus and the mathematical engine behind almost every learning algorithm in modern artificial intelligence. Training a neural network with billions of parameters comes down to computing one partial derivative per parameter, billions of times per second, and using those numbers to nudge each weight in the direction that lowers the loss. That this is even affordable rests on a classic complexity result: Walter Baur and Volker Strassen proved in 1983 that a function and all of its partial derivatives can be evaluated together for at most a small constant factor more than the cost of the function alone, independent of how many inputs there are ^[12]. This "cheap gradient" principle is the reason backpropagation can train models with hundreds of billions of parameters.

Formal definition

Let $f: \mathbb{R}^n \to \mathbb{R}$ be a real-valued function of n variables. The partial derivative of f with respect to $x_i$ at a point $x = (x_1, \ldots, x_n)$ is defined as the limit

$$\frac{\partial f}{\partial x_i}(x) = \lim_{h \to 0} \frac{f(x_1, \ldots, x_i + h, \ldots, x_n) - f(x_1, \ldots, x_i, \ldots, x_n)}{h}$$

when this limit exists. Geometrically, the partial derivative is the slope of the tangent line to the graph of f along the i-th coordinate axis, with all other coordinates frozen. If you slice the surface $z = f(x, y)$ with a plane $y = y_0$, you get a one-dimensional curve in x; the slope of that curve at $x_0$ is exactly $\frac{\partial f}{\partial x}$ evaluated at $(x_0, y_0)$.

This "freeze everything else" trick reduces a multivariable derivative to an ordinary single-variable derivative. To compute $\frac{\partial f}{\partial x_i}$ symbolically, treat every other variable as a constant and apply the usual rules of differentiation (power rule, product rule, quotient rule, chain rule).

Notation

Several notations for partial derivatives appear across textbooks, papers, and code. They all mean the same thing.

Scroll sideways for more →

The symbol ∂ is a stylized cursive d, sometimes called "del" or "partial", and it signals that the derivative is partial rather than total. A total derivative $df/dx$ accounts for indirect dependencies; a partial derivative does not.

Higher-order partial derivatives

Applying $\frac{\partial}{\partial x_j}$ to a partial derivative $\frac{\partial f}{\partial x_i}$ gives a second-order partial derivative

$$\frac{\partial^2 f}{\partial x_j \partial x_i}$$

When $i = j$, this is the pure second partial $\frac{\partial^2 f}{\partial x_i^2}$. When $i \ne j$, it is a mixed partial. A natural question is whether the order of differentiation matters. In general it can, but for well-behaved functions it does not.

Schwarz's theorem (Clairaut's theorem): If f and its first and second partial derivatives are continuous on an open neighborhood of a point, then

$$\frac{\partial^2 f}{\partial x_i \partial x_j} = \frac{\partial^2 f}{\partial x_j \partial x_i}$$

at that point. This result is sometimes attributed to Alexis Clairaut, who used it in 1740, or to Hermann Schwarz, who gave the first fully rigorous proof in 1873; Leonhard Euler had already relied on the same symmetry in 1740 ^[11]. It fails for pathological functions whose mixed partials exist but are discontinuous; Peano constructed a classical counterexample. Almost every loss function used in practice satisfies the smoothness assumptions, so deep learning practitioners rely on the symmetry of mixed partials without comment.

The gradient, Jacobian, and Hessian

Partial derivatives are usually packaged into matrices and vectors that summarize all first- or second-order information about a function.

Scroll sideways for more →

The gradient $\nabla f$ points in the direction in which f increases most rapidly, and its negative $-\nabla f$ gives the steepest descent direction. The Jacobian generalizes this to vector-valued functions: each row is the gradient of one output component. The Hessian, by Schwarz's theorem, is symmetric for any twice continuously differentiable function. Its eigenvalues tell you about the local curvature of the loss landscape, which connects partial derivatives to convexity and to the choice of learning rate.

The chain rule for multivariable functions

The chain rule extends to functions of several variables and is the workhorse of backpropagation. Suppose $y = f(u_1, \ldots, u_m)$ where each $u_k = g_k(x_1, \ldots, x_n)$. Then

$$\frac{\partial y}{\partial x_i} = \sum_k \frac{\partial f}{\partial u_k} \frac{\partial u_k}{\partial x_i}$$

This sum-of-products structure looks innocuous, but when you stack hundreds of layers it becomes a long matrix product. Each layer of a neural network contributes one Jacobian to the chain, and the gradient of the loss with respect to a weight deep inside the network is computed by multiplying these Jacobians together in a specific order. See the article on the chain rule for a deeper treatment.

Why do partial derivatives matter for machine learning?

A modern loss function L is a scalar-valued function of every learnable parameter in the model. A small language model has hundreds of millions of parameters; a frontier model has hundreds of billions. The fundamental learning step is

$$w_i \leftarrow w_i - \eta \cdot \frac{\partial L}{\partial w_i}$$

where $\eta$ is the learning rate. This is just gradient descent written one parameter at a time. The partial derivative $\frac{\partial L}{\partial w_i}$ answers a very specific question: if I increase weight $w_i$ by a tiny amount and leave every other weight alone, how much does the training loss change? The optimizer uses that answer to decide how to update each weight.

Making this practical requires three things: a way to compute ∂L/∂w_i for every parameter at once, a way to do it efficiently, and a way to do it without writing the formulas by hand. Backpropagation provides all three.

How does backpropagation compute partial derivatives efficiently?

The backpropagation algorithm, popularized by David Rumelhart, Geoffrey Hinton, and Ronald Williams in their 1986 Nature paper Learning representations by back-propagating errors, is a particular schedule for applying the multivariable chain rule. Their one-page paper opens: "We describe a new learning procedure, back-propagation, for networks of neurone-like units" ^[1]. Rather than recomputing the product of Jacobians from scratch for each parameter, backprop traverses the network once forward to compute activations, then once backward to accumulate partial derivatives layer by layer.

The key efficiency insight is reuse. The partial derivative of the loss with respect to a hidden activation appears in the chain-rule expansion for every parameter that influences that activation. Backpropagation computes it once and shares it. The total cost of evaluating the gradient is a small constant times the cost of evaluating the loss itself, regardless of how many parameters there are. This is the cheap gradient principle established by Baur and Strassen in 1983: computing a scalar function together with all of its partial derivatives costs at most a small constant factor more than computing the function alone, a bound commonly quoted as three to five times in standard arithmetic cost models and independent of the number of inputs ^[12]. This complexity result is what makes deep learning computationally tractable; without it, training a billion-parameter model would be hopeless.

Automatic differentiation

Backpropagation is a special case of a more general technique called automatic differentiation, often shortened to autodiff or AD. Autodiff systematically applies the chain rule to any program built from differentiable primitives. A 2017 survey by Atilim Gunes Baydin, Barak Pearlmutter, Alexey Radul, and Jeffrey Siskind in the Journal of Machine Learning Research gives the standard reference, defining autodiff as "a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs" ^[2]. Autodiff comes in two main modes.

Scroll sideways for more →

For a function $f: \mathbb{R}^n \to \mathbb{R}^m$, forward-mode AD computes one column of the Jacobian per pass, and reverse-mode AD computes one row per pass. A neural network loss has $m = 1$ (a scalar) and n in the millions or billions, which makes reverse mode dramatically cheaper. That is why every major deep learning framework defaults to reverse mode.

The big three frameworks each expose autodiff slightly differently:

• PyTorch builds a dynamic computation graph as operations execute on tensors with requires_grad=True. Calling .backward() on a scalar walks the graph in reverse and populates the .grad attribute on every leaf tensor ^[8].
• TensorFlow records operations onto a tf.GradientTape context. Calling tape.gradient(loss, variables) replays the tape backward, applying the chain rule ^[9].
• JAX provides functional transformations: jax.grad for the gradient of a scalar function, jax.jvp for forward-mode Jacobian-vector products, and jax.vjp for reverse-mode vector-Jacobian products. Higher-order derivatives compose by repeated application ^[10].

All three implementations are exact up to floating-point rounding. They are not numerical approximations.

How does autodiff differ from numerical and symbolic differentiation?

Automatic differentiation is one of three ways a computer can compute partial derivatives. The other two are worth mentioning because both have failure modes that make them poor choices for neural network training.

Numerical differentiation uses finite differences:

$$\frac{\partial f}{\partial x_i} \approx \frac{f(x + h e_i) - f(x)}{h}$$

where e_i is the i-th standard basis vector. This is mathematically clean but computationally awful for ML. Each gradient component requires its own forward pass, so the cost is O(n) full evaluations of the loss; for n in the billions, this is hopeless. Worse, finite differences have a numerical accuracy problem: a small h gives small truncation error $h^2/6 \cdot M$ but large round-off error roughly $\epsilon/h$, where $\epsilon$ is machine epsilon. The two errors trade off, and the optimal h depends on the function. For double-precision floats, the best achievable accuracy is around $10^{-8}$, far worse than autodiff. Finite differences remain useful as a sanity check, sometimes called a "gradient check", to catch bugs in custom autodiff implementations ^[2].

Symbolic differentiation manipulates expressions algebraically, the way Mathematica or SymPy do. It produces exact closed-form derivatives. The downside is expression swell: differentiating a nested expression naively can produce derivative expressions exponentially larger than the original, because the chain rule duplicates subexpressions. Symbolic systems can mitigate this with common subexpression elimination, but for the deeply nested compositions that define neural networks, automatic differentiation is more practical.

Worked examples in machine learning

Linear regression

Linear regression with mean squared error has loss

$$L(w, b) = \frac{1}{2N} \sum_i (w \cdot x_i + b - y_i)^2$$

The partial derivative with respect to weight w_j is

$$\frac{\partial L}{\partial w_j} = \frac{1}{N} \sum_i (w \cdot x_i + b - y_i) \cdot x_{i,j}$$

and with respect to the bias is

$$\frac{\partial L}{\partial b} = \frac{1}{N} \sum_i (w \cdot x_i + b - y_i)$$

In vector form, the gradient with respect to w is $\frac{1}{N} X^\top (Xw + b - y)$. Setting this to zero recovers the closed-form normal equations. Most modern ML uses gradient descent rather than the normal equations because matrix inversion does not scale to large data.

Softmax cross-entropy

For a classifier with logits $z = Wh + b$, softmax probabilities $p_k = \exp(z_k) / \sum_j \exp(z_j)$, and cross-entropy loss $L = -\sum_k y_k \log p_k$ against a one-hot label $y$, the partial derivative with respect to the logits has a famously clean form:

$$\frac{\partial L}{\partial z_k} = p_k - y_k$$

The gradient is just "predicted probability minus target". This identity is the reason softmax and cross-entropy are almost always implemented as a single fused operation in deep learning libraries: the combined form is numerically stable and the derivative collapses to a subtraction. By the chain rule, $\frac{\partial L}{\partial W_{kj}} = (p_k - y_k) \cdot h_j$, which is what backprop pushes back to the previous layer.

ReLU and the subgradient at zero

The rectified linear unit ReLU$(x) = \max(0, x)$ has derivative 1 for $x > 0$ and 0 for $x < 0$, but it is not differentiable at $x = 0$ because the left and right derivatives disagree. In practice, this is harmless. ReLU is convex, so it has a well-defined subdifferential at zero, which is the closed interval $[0, 1]$. Any value in that interval is a valid subgradient and produces a correct subgradient descent step. PyTorch and TensorFlow both implement the convention $\partial\,\mathrm{ReLU}/\partial x \big|_{x=0} = 0$, picking the lower endpoint of the subdifferential ^[8]. Floating-point inputs hit exactly zero with vanishing probability anyway, so the choice almost never affects training in practice.

Practical issues in deep learning

Computing partial derivatives for a deep network exposes several numerical and statistical pitfalls.

Vanishing gradients. When a network is many layers deep, the chain rule multiplies a long sequence of Jacobians. If most of those Jacobians have spectral norm less than one, the product shrinks exponentially with depth, and the gradient at early layers becomes too small to drive learning. Sepp Hochreiter identified this in his 1991 diploma thesis ^[3], and Yoshua Bengio and collaborators analyzed it again in 1994 ^[4]. The vanishing gradient problem motivated LSTM cells for recurrent networks, residual connections in ResNets, and careful weight initialization.

Exploding gradients. The opposite failure mode happens when Jacobian norms exceed one and the product grows exponentially. The standard fix, introduced by Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio in their 2013 paper on training recurrent networks, is gradient clipping: rescale the gradient vector to a fixed maximum norm before applying the update ^[5]. Modern transformer training routinely clips gradients to a global norm of 1.0; for example, GPT-3 was trained with Adam ($\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$) and the global norm of the gradient clipped at 1.0 ^[13].

Mixed-precision training. Float16 has only about three significant decimal digits, so partial derivatives that underflow to zero in fp16 can derail training. The standard workaround, introduced by Paulius Micikevicius and colleagues in 2018, is loss scaling: multiply the loss by a large constant before backprop, then divide gradients by the same constant ^[14]. The bfloat16 format keeps the same dynamic range as fp32 (both use 8 exponent bits) and largely sidesteps the issue, which is why bf16 has become the default for large model training ^[15].

Stop-gradient operators. Sometimes you want to use a tensor as a constant during backprop even though it depends on parameters. PyTorch provides tensor.detach(), TensorFlow provides tf.stop_gradient, and JAX provides jax.lax.stop_gradient. These primitives appear in policy-gradient methods (where the value function is treated as fixed when computing the policy update), in Gumbel-softmax estimators, in target networks for deep Q-learning, and in self-supervised methods like BYOL where one branch's gradient must be blocked.

Implicit differentiation

Not every quantity in modern ML is defined by an explicit forward computation. A deep equilibrium model, introduced by Shaojie Bai, J. Zico Kolter, and Vladlen Koltun in 2019, defines its hidden state as the fixed point of a nonlinear map $z^* = f(z^*, x)$. To compute $\frac{\partial z^*}{\partial x}$ without storing every iteration of the fixed-point solver, the model uses implicit differentiation derived from the implicit function theorem ^[6]. The same trick appears in implicit MAML for meta-learning, where Aravind Rajeswaran and collaborators differentiate through the optimality condition of an inner-loop optimizer instead of differentiating through the optimizer steps themselves ^[7]. Implicit differentiation gives constant-memory backward passes, which is otherwise impossible for problems with arbitrarily deep computation.

Explain Like I'm 5 (ELI5)

Imagine you're on a hill with lots of bumps and valleys, and you're trying to find the lowest point on the hill. To do this, you can only take one step at a time, and you want each step to take you lower. This is like what happens in machine learning when we try to make our model better by finding the lowest point of a special hill, called the "loss function."

Partial derivatives are like a magical compass that shows us which way to take a step to go lower on the hill. They tell us how the height of the hill changes if we only move in one direction. By following the directions given by the partial derivatives, we can find the lowest point on the hill, which helps make our machine learning model better.

References

1. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors." *Nature*, 323(6088), 533-536. https://www.nature.com/articles/323533a0 ↩
2. Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2017). "Automatic differentiation in machine learning: a survey." *Journal of Machine Learning Research*, 18(153), 1-43. https://www.jmlr.org/papers/v18/17-468.html ↩
3. Hochreiter, S. (1991). "Untersuchungen zu dynamischen neuronalen Netzen." Diploma thesis, Technische Universität München. ↩
4. Bengio, Y., Simard, P., & Frasconi, P. (1994). "Learning long-term dependencies with gradient descent is difficult." *IEEE Transactions on Neural Networks*, 5(2), 157-166. ↩
5. Pascanu, R., Mikolov, T., & Bengio, Y. (2013). "On the difficulty of training recurrent neural networks." *Proceedings of the 30th International Conference on Machine Learning*. https://arxiv.org/abs/1211.5063 ↩
6. Bai, S., Kolter, J. Z., & Koltun, V. (2019). "Deep Equilibrium Models." *Advances in Neural Information Processing Systems 32*. https://arxiv.org/abs/1909.01377 ↩
7. Rajeswaran, A., Finn, C., Kakade, S., & Levine, S. (2019). "Meta-Learning with Implicit Gradients." *Advances in Neural Information Processing Systems 32*. https://arxiv.org/abs/1909.04630 ↩
8. PyTorch documentation. "Automatic differentiation package, torch.autograd." https://docs.pytorch.org/docs/stable/autograd.html ↩
9. TensorFlow documentation. "Introduction to gradients and automatic differentiation." https://www.tensorflow.org/guide/autodiff ↩
10. JAX documentation. "The Autodiff Cookbook." https://docs.jax.dev/en/latest/notebooks/autodiff_cookbook.html ↩
11. Wikipedia. "Symmetry of second derivatives." https://en.wikipedia.org/wiki/Symmetry_of_second_derivatives ↩
12. Baur, W., & Strassen, V. (1983). "The complexity of partial derivatives." *Theoretical Computer Science*, 22(3), 317-330. https://www.sciencedirect.com/science/article/pii/030439758390110X ↩
13. Brown, T. B., et al. (2020). "Language Models are Few-Shot Learners." *Advances in Neural Information Processing Systems 33*. https://arxiv.org/abs/2005.14165 ↩
14. Micikevicius, P., et al. (2018). "Mixed Precision Training." *International Conference on Learning Representations (ICLR)*. https://arxiv.org/abs/1710.03740 ↩
15. Google Cloud. "BFloat16: The secret to high performance on Cloud TPUs." https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus ↩