# Ridge Regularization > Source: https://aiwiki.ai/wiki/ridge_regularization > Updated: 2026-04-27 > Categories: Machine Learning, Statistics, Training & Optimization > From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution. Ridge regularization, also known as **L2 regularization** or **Tikhonov regularization**, is a technique in [statistics](/wiki/bias_math_or_bias_term) and [machine learning](/wiki/machine_learning) that adds a squared L2-norm penalty to a model's [loss function](/wiki/loss_function). By penalizing large coefficient values, ridge regularization shrinks parameter estimates toward zero without setting them exactly to zero. This shrinkage reduces model [variance](/wiki/bias) at the cost of introducing a small amount of bias, which typically leads to better generalization on unseen data. The method is widely used in [linear regression](/wiki/linear_regression), [logistic regression](/wiki/logistic_regression), [neural networks](/wiki/neural_network), and many other statistical models. ## Explain like I'm 5 (ELI5) Imagine you are building a tower out of blocks, and you want it to be as tall as possible. If you stack all your blocks in one wobbly column, the tower might be very tall but it will fall over easily. Ridge regularization is like a rule that says "no single column of blocks can be too tall." So instead of one giant wobbly tower, you spread your blocks out more evenly and build a shorter but much sturdier structure. You give up a little bit of height (accuracy on your training data), but your tower stays standing even when someone bumps the table (new data comes in). In technical terms, ridge regularization tells your model: "You can fit the data, but you are not allowed to make any single weight too large." This keeps the model from relying too heavily on any one feature, making its predictions more stable and reliable. ## History The mathematical foundations of ridge regularization trace back to the Soviet mathematician Andrey Tikhonov, who published "On the stability of inverse problems" in 1943. Tikhonov developed the technique to solve ill-posed inverse problems in mathematical physics, where small perturbations in the input data could produce wildly different solutions. His approach added a penalty term to stabilize the solution. Independently, Arthur Hoerl and Robert Kennard introduced the statistical version of the method in 1970 through two papers published in the journal *Technometrics*: "Ridge Regressions: Biased Estimation for Nonorthogonal Problems" and "Ridge Regressions: Applications to Nonorthogonal Problems." Hoerl, a statistician at DuPont, had first proposed the idea as early as 1962 as a way to control the instability in [least squares](/wiki/least_squares_regression) estimates when predictor variables are correlated. The name "ridge regression" comes from ridge analysis, referring to the path from a constrained maximum along a ridge of the response surface. Around the same period, David L. Phillips applied similar regularization ideas to integral equations, and Manus Foster interpreted the method as a Wiener-Kolmogorov (Kriging) filter. These independent discoveries across different fields highlight the generality and importance of the L2 penalty concept. ## Mathematical formulation ### Problem setup Consider a standard [supervised learning](/wiki/supervised_machine_learning) problem with a [training set](/wiki/training_set) of *n* observations. Let **X** be the *n* x *p* design matrix (where each row is a training example and each column is a [feature](/wiki/feature)), **y** be the *n*-dimensional target vector, and **w** be the *p*-dimensional vector of model [weights](/wiki/weight). In ordinary least squares (OLS) regression, the goal is to minimize the residual sum of squares: **min_w ||y - Xw||^2** The OLS solution is: **w_OLS = (X^T X)^(-1) X^T y** This solution exists only when X^T X is invertible (i.e., X has full column rank). When predictors are highly correlated ([multicollinearity](/wiki/correlation)), X^T X becomes nearly singular, and the OLS estimates become unstable with large variances. ### Ridge regression objective Ridge regression modifies the OLS objective by adding an L2 penalty term: **min_w ||y - Xw||^2 + lambda * ||w||^2** where: - ||y - Xw||^2 is the residual sum of squares (data fidelity term) - ||w||^2 = w^T w is the squared L2 norm of the coefficient vector - lambda >= 0 is the [regularization](/wiki/regularization) parameter (also called the ridge parameter or penalty parameter) The first term ensures the model fits the training data, while the second term penalizes large coefficient values. The parameter lambda controls the balance between these two objectives. When lambda = 0, the solution reduces to OLS. As lambda increases, the coefficients are shrunk more aggressively toward zero. ### Closed-form solution Taking the gradient of the ridge objective with respect to **w** and setting it to zero: **gradient = -2 X^T (y - Xw) + 2 lambda w = 0** Solving for **w** yields the closed-form solution: **w_ridge = (X^T X + lambda I)^(-1) X^T y** where **I** is the *p* x *p* identity matrix. The addition of lambda I to X^T X guarantees that the matrix is positive definite for any lambda > 0, even when X^T X is singular or nearly singular. This means the ridge solution always exists and is unique, unlike the OLS solution which requires X to have full column rank. ### Constrained optimization form The ridge regression problem can equivalently be written as a constrained optimization problem: **min_w ||y - Xw||^2 subject to ||w||^2 <= c** where *c* is a constant that has a one-to-one relationship with lambda through the Lagrange multiplier formulation. Smaller values of *c* correspond to larger values of lambda, imposing tighter constraints on the coefficient magnitudes. ### General Tikhonov regularization The general form of Tikhonov regularization replaces the identity matrix with a regularization matrix Gamma: **min_x ||Ax - b||^2 + ||Gamma x||^2** The solution is: **x_hat = (A^T A + Gamma^T Gamma)^(-1) A^T b** When Gamma = sqrt(lambda) * I, this reduces to standard ridge regression. Choosing different Gamma matrices allows incorporating prior knowledge about the expected smoothness or structure of the solution. ## Singular value decomposition interpretation The behavior of ridge regression can be understood through the [singular value decomposition](/wiki/dimension_reduction) (SVD) of the design matrix. Let X = U D V^T, where: - U is an *n* x *n* orthogonal matrix (left singular vectors) - D is an *n* x *p* diagonal matrix with singular values d_1 >= d_2 >= ... >= d_p >= 0 - V is a *p* x *p* orthogonal matrix (right singular vectors, which are the principal component directions) Using the SVD, the ridge estimator can be written as: **w_ridge = V diag(d_j^2 / (d_j^2 + lambda)) D^(-1) U^T y** Compared to the OLS estimator w_OLS = V D^(-1) U^T y, the ridge estimator multiplies each component by a shrinkage factor: **f_j = d_j^2 / (d_j^2 + lambda)** This shrinkage factor lies between 0 and 1. Directions corresponding to large singular values (where the data has high variance) are shrunk very little, since d_j^2 >> lambda. Directions corresponding to small singular values (where the data has low variance) are shrunk heavily, since d_j^2 << lambda. This differential shrinkage is the mechanism by which ridge regression stabilizes the estimation: it aggressively dampens the noisy, poorly-determined directions while preserving the well-determined ones. When X^T X is singular (one or more singular values are zero), OLS cannot invert the matrix. Ridge regression effectively shifts all eigenvalues upward by lambda, transforming the singular matrix into a non-singular one. ## Bias-variance tradeoff Ridge regression is a classic example of the [bias-variance tradeoff](/wiki/bias_variance_tradeoff) in statistical estimation. ### Bias of the ridge estimator The expected value of the ridge estimator (conditional on X) under the true model y = Xw_true + epsilon is: **E[w_ridge | X] = (X^T X + lambda I)^(-1) X^T X w_true** The bias is: **Bias = E[w_ridge] - w_true = -lambda (X^T X + lambda I)^(-1) w_true** The bias is non-zero whenever lambda > 0 and w_true != 0. The ridge estimator systematically underestimates the true coefficients by shrinking them toward zero. Larger values of lambda produce greater bias. ### Variance of the ridge estimator The covariance matrix of the ridge estimator is: **Var(w_ridge | X) = sigma^2 (X^T X + lambda I)^(-1) X^T X (X^T X + lambda I)^(-1)** The variance of the ridge estimator is always smaller than the variance of the OLS estimator. The difference Var(w_OLS) - Var(w_ridge) is positive semi-definite for all lambda > 0. ### Mean squared error The mean squared error (MSE) of an estimator combines both bias and variance: **MSE = Variance + Bias^2** A theorem proven by Theobald (1974) and Farebrother (1976) states that there always exists a value of lambda such that the ridge estimator has lower MSE than the OLS estimator. In other words, even though OLS is unbiased, the reduction in variance from ridge regularization more than compensates for the introduced bias at the right penalty level. At small lambda, the estimator has low bias but high variance (close to OLS). At large lambda, the estimator has high bias but low variance (coefficients shrunk heavily toward zero). The optimal lambda balances these two effects to minimize total error. ## Effective degrees of freedom In OLS regression with *p* predictors, the model has exactly *p* degrees of freedom. Ridge regression has a continuous, reduced effective degrees of freedom given by: **df(lambda) = tr(H) = sum_{j=1}^{p} d_j^2 / (d_j^2 + lambda)** where H = X(X^T X + lambda I)^(-1) X^T is the ridge hat matrix, and d_j are the singular values of X. When lambda = 0, df = p (the OLS case). As lambda increases, the effective degrees of freedom decreases continuously toward zero. This quantity is useful for model comparison, information criteria (such as AIC or BIC), and generalized cross-validation. ## Geometric interpretation The constrained form of ridge regression has an appealing geometric interpretation. In two dimensions (two coefficients w_1 and w_2), the constraint ||w||^2 <= c defines a circular region centered at the origin. The OLS loss function defines elliptical contours around the unconstrained OLS solution. The ridge solution is the point where the smallest elliptical contour first touches the circular constraint region. Because the circle has a smooth, curved boundary with no corners, this tangent point almost always occurs at a location where both coefficients are non-zero. This is why ridge regression shrinks coefficients toward zero but rarely sets them exactly to zero. This contrasts with [L1 regularization](/wiki/l1_regularization) (lasso), where the constraint region is a diamond shape with sharp corners at the axes. The lasso's angular constraint region makes it much more likely that the tangent point falls on a corner, resulting in one or more coefficients being exactly zero. This fundamental geometric difference explains why lasso performs [feature selection](/wiki/feature_engineering) while ridge does not. In higher dimensions, the ridge constraint region is a hypersphere, while the lasso constraint region is a cross-polytope (hyperoctahedron). ## Choosing the regularization parameter The regularization parameter lambda determines the strength of the penalty and must be selected carefully. Several methods exist for choosing lambda. ### Cross-validation [Cross-validation](/wiki/cross-validation) is the most common approach. In *k*-fold cross-validation: 1. Divide the training data into *k* roughly equal folds. 2. For each candidate value of lambda, train the model on *k - 1* folds and evaluate on the held-out fold. 3. Average the prediction error across all *k* folds. 4. Select the lambda that minimizes the average prediction error. Leave-one-out cross-validation (LOOCV) is a special case where *k = n*. For ridge regression, LOOCV can be computed efficiently in closed form without actually refitting the model *n* times, thanks to the Sherman-Morrison-Woodbury formula. ### Generalized cross-validation (GCV) Proposed by Golub, Heath, and Wahba (1979), GCV is a rotation-invariant approximation to leave-one-out cross-validation. The GCV score for a given lambda is: **GCV(lambda) = (1/n) ||y - X w_ridge||^2 / [1 - tr(H)/n]^2** where tr(H) is the trace of the hat matrix (i.e., the effective degrees of freedom). GCV does not require an estimate of the noise variance sigma^2, which makes it practical when the noise level is unknown. The optimal lambda minimizes the GCV score. ### Information criteria AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) can be adapted for ridge regression using the effective degrees of freedom df(lambda) in place of the number of parameters. ### Bayesian approach From a Bayesian perspective (see section below), lambda can be treated as a [hyperparameter](/wiki/hyperparameter) and estimated from the data using empirical Bayes or marginal likelihood maximization. ## Bayesian interpretation Ridge regression has a natural Bayesian interpretation. Suppose we place a multivariate normal prior on the weight vector: **w ~ N(0, tau^2 I)** and assume the likelihood is: **y | X, w ~ N(Xw, sigma^2 I)** Then the maximum a posteriori (MAP) estimate of w is exactly the ridge regression solution with lambda = sigma^2 / tau^2. In this view, the regularization parameter lambda encodes the ratio of noise variance to prior variance. A large lambda (strong regularization) corresponds to a prior that strongly concentrates the weights near zero. A small lambda (weak regularization) corresponds to a diffuse prior that allows the weights to take on large values. The full Bayesian treatment goes beyond the MAP estimate to compute the entire posterior distribution of the weights, which is also Gaussian: **w | y, X ~ N(w_ridge, (X^T X / sigma^2 + I / tau^2)^(-1))** This posterior provides uncertainty estimates for all coefficients, which can be useful for constructing credible intervals and making probabilistic predictions. ## Scale dependence and standardization Ridge regression is not scale-invariant. Rescaling a predictor variable by a constant changes the relative magnitude of its coefficient and therefore changes the effect of the penalty. For example, if one predictor is measured in meters and another in kilometers, the coefficient for the meters-based predictor will be much smaller, and ridge regression will penalize it less. To ensure fair penalization across all features, practitioners typically standardize the predictors before applying ridge regression. Standardization involves subtracting the mean and dividing by the standard deviation of each feature, so that all features have zero mean and unit variance. The intercept term is usually not penalized, as its scale depends on the target variable rather than on the feature scales. ## Comparison with other regularization methods Ridge regularization belongs to a family of penalized regression methods. The following table summarizes the differences among the most common approaches. | Property | Ridge (L2) | [Lasso](/wiki/l1_regularization) (L1) | [Elastic net](/wiki/regularization) (L1 + L2) | |---|---|---|---| | Penalty term | lambda * sum(w_j^2) | lambda * sum(abs(w_j)) | lambda_1 * sum(abs(w_j)) + lambda_2 * sum(w_j^2) | | Produces sparse models | No | Yes | Yes | | Feature selection | No | Yes | Yes | | Handles multicollinearity | Yes, shrinks correlated coefficients together | Unstable, may arbitrarily select one of correlated features | Yes, selects groups of correlated features | | Closed-form solution | Yes | No (requires iterative methods) | No (requires iterative methods) | | Geometric constraint shape | Hypersphere | Cross-polytope (diamond) | Blend of sphere and diamond | | When to use | Many features with similar importance | Few truly relevant features among many irrelevant ones | Large feature sets with groups of correlated features | Ridge regression tends to perform well when most features contribute to the prediction and their true coefficients are of similar magnitude. Lasso is preferred when the true model is sparse (only a few features matter). Elastic net combines the strengths of both and is often a good default when the underlying sparsity pattern is unknown. ## Ridge regularization in neural networks (weight decay) In [deep learning](/wiki/deep_neural_network), L2 regularization is commonly referred to as **weight decay**. The idea is the same: a penalty proportional to the sum of squared weights is added to the loss function: **Loss_total = Loss_data + lambda * sum(w_ij^2)** where the sum runs over all weights in the network. This encourages the network to distribute information across many neurons rather than concentrating it in a few large weights, which reduces [overfitting](/wiki/overfitting). ### Weight decay vs. L2 regularization For standard [stochastic gradient descent](/wiki/stochastic_gradient_descent_sgd) (SGD), weight decay and L2 regularization produce identical parameter updates. The gradient of the L2 penalty adds a term proportional to the current weight at each step, which is equivalent to multiplying each weight by a factor slightly less than 1 before applying the gradient update. However, for adaptive optimizers like [Adam](/wiki/optimizer), the two approaches are not equivalent. L2 regularization adds the penalty gradient to the loss gradient before the adaptive scaling step, which means that parameters with large historical gradients receive weaker effective regularization. Loshchilov and Hutter (2019) showed that decoupling the weight decay from the gradient adaptation step (a variant called AdamW) produces more consistent regularization and often leads to better performance. In AdamW, the weight decay is applied directly to the weights after the gradient update, rather than being routed through the adaptive gradient mechanism. ### Practical considerations Weight decay is typically applied to all learnable weight matrices in a neural network. Bias terms and [batch normalization](/wiki/batch_normalization) parameters are usually excluded from the penalty, as regularizing them can interfere with the network's ability to shift and scale activations. The lambda value in deep learning is often set to small values like 1e-4 or 1e-5, though the optimal value depends on the architecture and dataset. ## Kernel ridge regression Kernel ridge regression (KRR) extends ridge regression to nonlinear problems by combining it with the kernel trick. Instead of learning a linear function in the original feature space, KRR learns a linear function in a high-dimensional (potentially infinite-dimensional) reproducing kernel Hilbert space (RKHS) induced by a kernel function. ### Dual formulation Ridge regression can be written in a dual form that depends on inner products between data points rather than the features themselves. The dual solution is: **alpha = (K + lambda I)^(-1) y** where K = X X^T is the *n* x *n* kernel matrix (Gram matrix) with entries K_ij = k(x_i, x_j), and the prediction for a new input x is: **f(x) = sum_{i=1}^{n} alpha_i k(x_i, x)** By replacing the linear kernel with a nonlinear kernel function (such as the Gaussian/RBF kernel, polynomial kernel, or others), KRR can model complex nonlinear relationships without explicitly computing the feature map. ### Comparison with support vector regression Kernel ridge regression and [support vector](/wiki/bias) regression (SVR) learn models of the same functional form but differ in their loss functions and resulting properties. | Property | Kernel ridge regression | Support vector regression | |---|---|---| | Loss function | Squared error | Epsilon-insensitive | | Solution type | Dense (uses all training points) | Sparse (uses only support vectors) | | Fitting method | Closed-form | Iterative (quadratic programming) | | Training speed | Faster for medium-sized datasets | Slower for medium-sized datasets | | Prediction speed | Slower (depends on all training points) | Faster (depends only on support vectors) | | Scalability | O(n^3) time, O(n^2) memory | Better for large n due to sparsity | KRR is preferred when the training set is small to medium (up to a few thousand samples) and fast training time is needed. SVR is preferred for larger datasets where the sparsity of the solution provides computational savings at prediction time. ## Applications Ridge regularization is used across a wide range of fields: - **Econometrics and social sciences:** Ridge regression stabilizes coefficient estimates in models with many correlated economic indicators, such as macroeconomic forecasting with dozens of interrelated variables. - **Genomics and bioinformatics:** In genome-wide association studies, the number of genetic features (SNPs) often far exceeds the number of samples. Ridge regression provides stable coefficient estimates in this high-dimensional setting. - **Chemistry and spectroscopy:** Spectral data from instruments often contains hundreds of highly correlated wavelength channels. Ridge regression (often via partial least squares, a related method) is standard practice for calibration models. - **[Computer vision](/wiki/computer_vision) and [natural language processing](/wiki/natural_language_understanding):** L2 regularization (weight decay) is a default setting in virtually all deep learning training pipelines for image classification, [object detection](/wiki/object_detection), [machine translation](/wiki/machine_translation), and other tasks. - **Signal processing:** Tikhonov regularization is used extensively in signal deconvolution, image reconstruction (such as in MRI and CT scanning), and inverse problems in geophysics. - **[Recommendation systems](/wiki/recommender_system):** Matrix factorization methods for collaborative filtering typically include L2 penalties on both user and item latent factor vectors to prevent overfitting on sparse interaction data. ## Implementation Ridge regression is available in most statistical and machine learning software libraries. | Library | Language | Class or function | |---|---|---| | [scikit-learn](/wiki/scikit-learn) | [Python](/wiki/data_analysis) | `sklearn.linear_model.Ridge`, `sklearn.kernel_ridge.KernelRidge` | | statsmodels | Python | `OLS` with `fit_regularized(alpha, L1_wt=0)` | | [PyTorch](/wiki/pytorch) | Python | `torch.optim.SGD(weight_decay=...)` or `torch.optim.AdamW` | | [TensorFlow](/wiki/tensorflow) / [Keras](/wiki/keras) | Python | `tf.keras.regularizers.L2(lambda)` | | glmnet | R | `glmnet(alpha=0)` | | MASS | R | `lm.ridge()` | | MATLAB | MATLAB | `ridge()` or `lasso(Alpha=0)` | In scikit-learn, the regularization parameter is called `alpha` rather than `lambda` (to avoid a conflict with the Python keyword `lambda`). The `RidgeCV` class provides built-in cross-validation for selecting the optimal alpha. ## Limitations - **No feature selection:** Ridge regression shrinks all coefficients toward zero but does not set any to exactly zero. When interpretability through feature selection is needed, lasso or elastic net is more appropriate. - **Scale dependence:** The penalty depends on the scale of the features, so standardization is required before fitting. - **Bias introduction:** The ridge estimator is biased, which can be undesirable when unbiased estimates are needed for inference or hypothesis testing. - **Linear assumption in standard form:** Standard ridge regression assumes a linear relationship between features and the target. Nonlinear extensions require kernel ridge regression or other approaches. - **Lambda selection overhead:** Finding the optimal regularization parameter requires cross-validation or similar procedures, adding computational cost. ## See also - [Regularization](/wiki/regularization) - [L1 regularization](/wiki/l1_regularization) - [L2 regularization](/wiki/l2_regularization) - [Overfitting](/wiki/overfitting) - [Bias-variance tradeoff](/wiki/bias_variance_tradeoff) - [Cross-validation](/wiki/cross-validation) - [Linear regression](/wiki/linear_regression) - [Loss function](/wiki/loss_function) - [Gradient descent](/wiki/stochastic_gradient_descent_sgd) ## References 1. Hoerl, A. E., & Kennard, R. W. (1970). "Ridge Regressions: Biased Estimation for Nonorthogonal Problems." *Technometrics*, 12(1), 55-67. 2. Hoerl, A. E., & Kennard, R. W. (1970). "Ridge Regressions: Applications to Nonorthogonal Problems." *Technometrics*, 12(1), 69-82. 3. Tikhonov, A. N. (1943). "On the stability of inverse problems." *Doklady Akademii Nauk SSSR*, 39(5), 195-198. 4. Golub, G. H., Heath, M., & Wahba, G. (1979). "Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter." *Technometrics*, 21(2), 215-223. 5. Theobald, C. M. (1974). "Generalizations of Mean Square Error Applied to Ridge Regression." *Journal of the Royal Statistical Society, Series B*, 36(1), 103-106. 6. Farebrother, R. W. (1976). "Further Results on the Mean Square Error of Ridge Regression." *Journal of the Royal Statistical Society, Series B*, 38(3), 248-250. 7. Hastie, T., Tibshirani, R., & Friedman, J. (2009). *The Elements of Statistical Learning: Data Mining, Inference, and Prediction.* 2nd ed. Springer. 8. Bishop, C. M. (2006). *Pattern Recognition and Machine Learning.* Springer. 9. Murphy, K. P. (2012). *Machine Learning: A Probabilistic Perspective.* MIT Press. 10. Loshchilov, I., & Hutter, F. (2019). "Decoupled Weight Decay Regularization." *Proceedings of the 7th International Conference on Learning Representations (ICLR)*. 11. Hoerl, R. W. (2020). "Ridge Regression: A Historical Context." *Technometrics*, 62(4), 420-425. 12. Pedregosa, F., et al. (2011). "Scikit-learn: Machine Learning in Python." *Journal of Machine Learning Research*, 12, 2825-2830. 13. van Wieringen, W. N. (2021). "Lecture notes on ridge regression." *arXiv preprint arXiv:1509.09169*. 14. Saunders, C., Gammerman, A., & Vovk, V. (1998). "Ridge Regression Learning Algorithm in Dual Variables." *Proceedings of the 15th International Conference on Machine Learning (ICML)*, 515-521.