CtrlK
Ridge regularization, also known as L2 regularization or Tikhonov regularization, is a technique in statistics and machine learning that adds a squared L2-norm penalty to a model's loss function. By penalizing large coefficient values, ridge regularization shrinks parameter estimates toward zero without setting them exactly to zero. This shrinkage reduces model variance at the cost of introducing a small amount of bias, which typically leads to better generalization on unseen data. The method is widely used in linear regression, logistic regression, neural networks, and many other statistical models.
Imagine you are building a tower out of blocks, and you want it to be as tall as possible. If you stack all your blocks in one wobbly column, the tower might be very tall but it will fall over easily. Ridge regularization is like a rule that says "no single column of blocks can be too tall." So instead of one giant wobbly tower, you spread your blocks out more evenly and build a shorter but much sturdier structure. You give up a little bit of height (accuracy on your training data), but your tower stays standing even when someone bumps the table (new data comes in).
In technical terms, ridge regularization tells your model: "You can fit the data, but you are not allowed to make any single weight too large." This keeps the model from relying too heavily on any one feature, making its predictions more stable and reliable.
The mathematical foundations of ridge regularization trace back to the Soviet mathematician Andrey Tikhonov, who published "On the stability of inverse problems" in 1943. Tikhonov developed the technique to solve ill-posed inverse problems in mathematical physics, where small perturbations in the input data could produce wildly different solutions. His approach added a penalty term to stabilize the solution.
Independently, Arthur Hoerl and Robert Kennard introduced the statistical version of the method in 1970 through two papers published in the journal Technometrics: "Ridge Regressions: Biased Estimation for Nonorthogonal Problems" and "Ridge Regressions: Applications to Nonorthogonal Problems." Hoerl, a statistician at DuPont, had first proposed the idea as early as 1962 as a way to control the instability in least squares estimates when predictor variables are correlated. The name "ridge regression" comes from ridge analysis, referring to the path from a constrained maximum along a ridge of the response surface.
Around the same period, David L. Phillips applied similar regularization ideas to integral equations, and Manus Foster interpreted the method as a Wiener-Kolmogorov (Kriging) filter. These independent discoveries across different fields highlight the generality and importance of the L2 penalty concept.
Consider a standard supervised learning problem with a training set of n observations. Let X be the n x p design matrix (where each row is a training example and each column is a feature), y be the n-dimensional target vector, and w be the p-dimensional vector of model weights.
In ordinary least squares (OLS) regression, the goal is to minimize the residual sum of squares:
min_w ||y - Xw||^2
The OLS solution is:
w_OLS = (X^T X)^(-1) X^T y
This solution exists only when X^T X is invertible (i.e., X has full column rank). When predictors are highly correlated (multicollinearity), X^T X becomes nearly singular, and the OLS estimates become unstable with large variances.
Ridge regression modifies the OLS objective by adding an L2 penalty term:
min_w ||y - Xw||^2 + lambda * ||w||^2
where:
The first term ensures the model fits the training data, while the second term penalizes large coefficient values. The parameter lambda controls the balance between these two objectives. When lambda = 0, the solution reduces to OLS. As lambda increases, the coefficients are shrunk more aggressively toward zero.
Taking the gradient of the ridge objective with respect to w and setting it to zero:
gradient = -2 X^T (y - Xw) + 2 lambda w = 0
Solving for w yields the closed-form solution:
w_ridge = (X^T X + lambda I)^(-1) X^T y
where I is the p x p identity matrix. The addition of lambda I to X^T X guarantees that the matrix is positive definite for any lambda > 0, even when X^T X is singular or nearly singular. This means the ridge solution always exists and is unique, unlike the OLS solution which requires X to have full column rank.
The ridge regression problem can equivalently be written as a constrained optimization problem:
min_w ||y - Xw||^2 subject to ||w||^2 <= c
where c is a constant that has a one-to-one relationship with lambda through the Lagrange multiplier formulation. Smaller values of c correspond to larger values of lambda, imposing tighter constraints on the coefficient magnitudes.
The general form of Tikhonov regularization replaces the identity matrix with a regularization matrix Gamma:
min_x ||Ax - b||^2 + ||Gamma x||^2
The solution is:
x_hat = (A^T A + Gamma^T Gamma)^(-1) A^T b
When Gamma = sqrt(lambda) * I, this reduces to standard ridge regression. Choosing different Gamma matrices allows incorporating prior knowledge about the expected smoothness or structure of the solution.
The behavior of ridge regression can be understood through the singular value decomposition (SVD) of the design matrix. Let X = U D V^T, where:
Using the SVD, the ridge estimator can be written as:
w_ridge = V diag(d_j^2 / (d_j^2 + lambda)) D^(-1) U^T y
Compared to the OLS estimator w_OLS = V D^(-1) U^T y, the ridge estimator multiplies each component by a shrinkage factor:
f_j = d_j^2 / (d_j^2 + lambda)
This shrinkage factor lies between 0 and 1. Directions corresponding to large singular values (where the data has high variance) are shrunk very little, since d_j^2 >> lambda. Directions corresponding to small singular values (where the data has low variance) are shrunk heavily, since d_j^2 << lambda. This differential shrinkage is the mechanism by which ridge regression stabilizes the estimation: it aggressively dampens the noisy, poorly-determined directions while preserving the well-determined ones.
When X^T X is singular (one or more singular values are zero), OLS cannot invert the matrix. Ridge regression effectively shifts all eigenvalues upward by lambda, transforming the singular matrix into a non-singular one.
Ridge regression is a classic example of the bias-variance tradeoff in statistical estimation.
The expected value of the ridge estimator (conditional on X) under the true model y = Xw_true + epsilon is:
E[w_ridge | X] = (X^T X + lambda I)^(-1) X^T X w_true
The bias is:
Bias = E[w_ridge] - w_true = -lambda (X^T X + lambda I)^(-1) w_true
The bias is non-zero whenever lambda > 0 and w_true != 0. The ridge estimator systematically underestimates the true coefficients by shrinking them toward zero. Larger values of lambda produce greater bias.
The covariance matrix of the ridge estimator is:
Var(w_ridge | X) = sigma^2 (X^T X + lambda I)^(-1) X^T X (X^T X + lambda I)^(-1)
The variance of the ridge estimator is always smaller than the variance of the OLS estimator. The difference Var(w_OLS) - Var(w_ridge) is positive semi-definite for all lambda > 0.
The mean squared error (MSE) of an estimator combines both bias and variance:
MSE = Variance + Bias^2
A theorem proven by Theobald (1974) and Farebrother (1976) states that there always exists a value of lambda such that the ridge estimator has lower MSE than the OLS estimator. In other words, even though OLS is unbiased, the reduction in variance from ridge regularization more than compensates for the introduced bias at the right penalty level.
At small lambda, the estimator has low bias but high variance (close to OLS). At large lambda, the estimator has high bias but low variance (coefficients shrunk heavily toward zero). The optimal lambda balances these two effects to minimize total error.
In OLS regression with p predictors, the model has exactly p degrees of freedom. Ridge regression has a continuous, reduced effective degrees of freedom given by:
df(lambda) = tr(H) = sum_{j=1}^{p} d_j^2 / (d_j^2 + lambda)
where H = X(X^T X + lambda I)^(-1) X^T is the ridge hat matrix, and d_j are the singular values of X.
When lambda = 0, df = p (the OLS case). As lambda increases, the effective degrees of freedom decreases continuously toward zero. This quantity is useful for model comparison, information criteria (such as AIC or BIC), and generalized cross-validation.
The constrained form of ridge regression has an appealing geometric interpretation. In two dimensions (two coefficients w_1 and w_2), the constraint ||w||^2 <= c defines a circular region centered at the origin. The OLS loss function defines elliptical contours around the unconstrained OLS solution.
The ridge solution is the point where the smallest elliptical contour first touches the circular constraint region. Because the circle has a smooth, curved boundary with no corners, this tangent point almost always occurs at a location where both coefficients are non-zero. This is why ridge regression shrinks coefficients toward zero but rarely sets them exactly to zero.
This contrasts with L1 regularization (lasso), where the constraint region is a diamond shape with sharp corners at the axes. The lasso's angular constraint region makes it much more likely that the tangent point falls on a corner, resulting in one or more coefficients being exactly zero. This fundamental geometric difference explains why lasso performs feature selection while ridge does not.
In higher dimensions, the ridge constraint region is a hypersphere, while the lasso constraint region is a cross-polytope (hyperoctahedron).
The regularization parameter lambda determines the strength of the penalty and must be selected carefully. Several methods exist for choosing lambda.
Cross-validation is the most common approach. In k-fold cross-validation:
Leave-one-out cross-validation (LOOCV) is a special case where k = n. For ridge regression, LOOCV can be computed efficiently in closed form without actually refitting the model n times, thanks to the Sherman-Morrison-Woodbury formula.
Proposed by Golub, Heath, and Wahba (1979), GCV is a rotation-invariant approximation to leave-one-out cross-validation. The GCV score for a given lambda is:
GCV(lambda) = (1/n) ||y - X w_ridge||^2 / [1 - tr(H)/n]^2
where tr(H) is the trace of the hat matrix (i.e., the effective degrees of freedom). GCV does not require an estimate of the noise variance sigma^2, which makes it practical when the noise level is unknown. The optimal lambda minimizes the GCV score.
AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) can be adapted for ridge regression using the effective degrees of freedom df(lambda) in place of the number of parameters.
From a Bayesian perspective (see section below), lambda can be treated as a hyperparameter and estimated from the data using empirical Bayes or marginal likelihood maximization.
Ridge regression has a natural Bayesian interpretation. Suppose we place a multivariate normal prior on the weight vector:
w ~ N(0, tau^2 I)
and assume the likelihood is:
y | X, w ~ N(Xw, sigma^2 I)
Then the maximum a posteriori (MAP) estimate of w is exactly the ridge regression solution with lambda = sigma^2 / tau^2.
In this view, the regularization parameter lambda encodes the ratio of noise variance to prior variance. A large lambda (strong regularization) corresponds to a prior that strongly concentrates the weights near zero. A small lambda (weak regularization) corresponds to a diffuse prior that allows the weights to take on large values.
The full Bayesian treatment goes beyond the MAP estimate to compute the entire posterior distribution of the weights, which is also Gaussian:
w | y, X ~ N(w_ridge, (X^T X / sigma^2 + I / tau^2)^(-1))
This posterior provides uncertainty estimates for all coefficients, which can be useful for constructing credible intervals and making probabilistic predictions.
Ridge regression is not scale-invariant. Rescaling a predictor variable by a constant changes the relative magnitude of its coefficient and therefore changes the effect of the penalty. For example, if one predictor is measured in meters and another in kilometers, the coefficient for the meters-based predictor will be much smaller, and ridge regression will penalize it less.
To ensure fair penalization across all features, practitioners typically standardize the predictors before applying ridge regression. Standardization involves subtracting the mean and dividing by the standard deviation of each feature, so that all features have zero mean and unit variance. The intercept term is usually not penalized, as its scale depends on the target variable rather than on the feature scales.
Ridge regularization belongs to a family of penalized regression methods. The following table summarizes the differences among the most common approaches.
| Property | Ridge (L2) | Lasso (L1) | Elastic net (L1 + L2) |
|---|---|---|---|
| Penalty term | lambda * sum(w_j^2) | lambda * sum(abs(w_j)) | lambda_1 * sum(abs(w_j)) + lambda_2 * sum(w_j^2) |
| Produces sparse models | No | Yes | Yes |
| Feature selection | No | Yes | Yes |
| Handles multicollinearity | Yes, shrinks correlated coefficients together | Unstable, may arbitrarily select one of correlated features | Yes, selects groups of correlated features |
| Closed-form solution | Yes | No (requires iterative methods) | No (requires iterative methods) |
| Geometric constraint shape | Hypersphere | Cross-polytope (diamond) | Blend of sphere and diamond |
| When to use | Many features with similar importance | Few truly relevant features among many irrelevant ones | Large feature sets with groups of correlated features |
Ridge regression tends to perform well when most features contribute to the prediction and their true coefficients are of similar magnitude. Lasso is preferred when the true model is sparse (only a few features matter). Elastic net combines the strengths of both and is often a good default when the underlying sparsity pattern is unknown.
In deep learning, L2 regularization is commonly referred to as weight decay. The idea is the same: a penalty proportional to the sum of squared weights is added to the loss function:
Loss_total = Loss_data + lambda * sum(w_ij^2)
where the sum runs over all weights in the network. This encourages the network to distribute information across many neurons rather than concentrating it in a few large weights, which reduces overfitting.
For standard stochastic gradient descent (SGD), weight decay and L2 regularization produce identical parameter updates. The gradient of the L2 penalty adds a term proportional to the current weight at each step, which is equivalent to multiplying each weight by a factor slightly less than 1 before applying the gradient update.
However, for adaptive optimizers like Adam, the two approaches are not equivalent. L2 regularization adds the penalty gradient to the loss gradient before the adaptive scaling step, which means that parameters with large historical gradients receive weaker effective regularization. Loshchilov and Hutter (2019) showed that decoupling the weight decay from the gradient adaptation step (a variant called AdamW) produces more consistent regularization and often leads to better performance. In AdamW, the weight decay is applied directly to the weights after the gradient update, rather than being routed through the adaptive gradient mechanism.
Weight decay is typically applied to all learnable weight matrices in a neural network. Bias terms and batch normalization parameters are usually excluded from the penalty, as regularizing them can interfere with the network's ability to shift and scale activations. The lambda value in deep learning is often set to small values like 1e-4 or 1e-5, though the optimal value depends on the architecture and dataset.
Kernel ridge regression (KRR) extends ridge regression to nonlinear problems by combining it with the kernel trick. Instead of learning a linear function in the original feature space, KRR learns a linear function in a high-dimensional (potentially infinite-dimensional) reproducing kernel Hilbert space (RKHS) induced by a kernel function.
Ridge regression can be written in a dual form that depends on inner products between data points rather than the features themselves. The dual solution is:
alpha = (K + lambda I)^(-1) y
where K = X X^T is the n x n kernel matrix (Gram matrix) with entries K_ij = k(x_i, x_j), and the prediction for a new input x is:
f(x) = sum_{i=1}^{n} alpha_i k(x_i, x)
By replacing the linear kernel with a nonlinear kernel function (such as the Gaussian/RBF kernel, polynomial kernel, or others), KRR can model complex nonlinear relationships without explicitly computing the feature map.
Kernel ridge regression and support vector regression (SVR) learn models of the same functional form but differ in their loss functions and resulting properties.
| Property | Kernel ridge regression | Support vector regression |
|---|---|---|
| Loss function | Squared error | Epsilon-insensitive |
| Solution type | Dense (uses all training points) | Sparse (uses only support vectors) |
| Fitting method | Closed-form | Iterative (quadratic programming) |
| Training speed | Faster for medium-sized datasets | Slower for medium-sized datasets |
| Prediction speed | Slower (depends on all training points) | Faster (depends only on support vectors) |
| Scalability | O(n^3) time, O(n^2) memory | Better for large n due to sparsity |
KRR is preferred when the training set is small to medium (up to a few thousand samples) and fast training time is needed. SVR is preferred for larger datasets where the sparsity of the solution provides computational savings at prediction time.
Ridge regularization is used across a wide range of fields:
Ridge regression is available in most statistical and machine learning software libraries.
| Library | Language | Class or function |
|---|---|---|
| scikit-learn | Python | sklearn.linear_model.Ridge, sklearn.kernel_ridge.KernelRidge |
| statsmodels | Python | OLS with fit_regularized(alpha, L1_wt=0) |
| PyTorch | Python | torch.optim.SGD(weight_decay=...) or torch.optim.AdamW |
| TensorFlow / Keras | Python | tf.keras.regularizers.L2(lambda) |
| glmnet | R | glmnet(alpha=0) |
| MASS | R | lm.ridge() |
| MATLAB | MATLAB | ridge() or lasso(Alpha=0) |
In scikit-learn, the regularization parameter is called alpha rather than lambda (to avoid a conflict with the Python keyword lambda). The RidgeCV class provides built-in cross-validation for selecting the optimal alpha.