Elastic Net

Elastic Net is a regularization method for linear regression that combines the L1 penalty associated with Lasso regression and the squared L2 penalty associated with ridge regression. With a positive L1 component, it can set some coefficients exactly to zero and therefore perform feature selection. Its positive L2 component can stabilize a fit when predictors are strongly correlated and can encourage correlated predictors to receive similar coefficients. Hui Zou and Trevor Hastie introduced the method in a 2005 paper.^[1]

Elastic Net is useful when a sparse model is desired but the predictors are not close to independent. It is especially associated with high-dimensional settings in which the number of candidate predictors is comparable to or greater than the number of observations. The method does not guarantee that every correlated variable will be selected, that selected variables are causally important, or that prediction will be better than both Lasso and Ridge. Those outcomes depend on the data, preprocessing, loss, and tuning procedure.^[1]

The name now has two related meanings. In contemporary statistics and machine learning, it usually refers to a loss minimized with a weighted sum of L1 and squared L2 penalties. In the original paper, "Elastic Net" referred more specifically to a rescaled version of what the authors called the naive Elastic Net. Most current software documentation defines the direct mixed-penalty objective and does not apply that original post-fit rescaling.^[1][7][8]

Definition

For a response vector (y), design matrix (X), intercept (\beta_0), and coefficient vector (\beta), a common modern Gaussian objective is

$$\min_{\beta_0,\beta} \frac{1}{2n}\sum_{i=1}^{n}(y_i-\beta_0-x_i^\top\beta)^2 +\lambda\left[ \alpha\lVert\beta\rVert_1+ \frac{1-\alpha}{2}\lVert\beta\rVert_2^2 \right].$$

Here (\lambda\geq 0) controls the overall penalty strength and (0\leq\alpha\leq 1) controls the mixture. The intercept is normally excluded from the penalty. With this convention:

• (\alpha=1) gives Lasso.
• (\alpha=0) gives Ridge.
• (0<\alpha<1) gives an Elastic Net penalty.
• (\lambda=0) removes both penalty terms, so the value of (\alpha) no longer affects the objective.

The L1 term is nondifferentiable at zero and can produce sparsity. The resulting fit is a convex optimization problem. If (\lambda>0) and (\alpha<1), the positive quadratic term makes the objective strictly convex in the coefficient vector. This gives a unique coefficient solution even when the columns of (X) are linearly dependent. By contrast, a Lasso solution can be nonunique for a rank-deficient design.^[1]

This formula is one parameterization, not a universal unit system. Multiplying the squared-error term, changing the definition of feature variance, standardizing the response, or using a different likelihood changes the numerical value that represents the same penalty. A (\lambda) copied from one library is therefore not necessarily equivalent in another.

Original parameterization and rescaling

Zou and Hastie first defined the naive estimator as

$$\hat{\beta}_{\mathrm{naive}} = \arg\min_{\beta} \left\{ \lVert y-X\beta\rVert_2^2 +\lambda_1\lVert\beta\rVert_1 +\lambda_2\lVert\beta\rVert_2^2 \right\},$$

after centering the response and standardizing the predictors. In that paper, the optional mixing quantity was

$$\alpha_{\mathrm{original}}= \frac{\lambda_2}{\lambda_1+\lambda_2}.$$

Its direction is opposite to the convention later adopted by glmnet: (\alpha_{\mathrm{original}}=0) is Lasso and (\alpha_{\mathrm{original}}=1) is Ridge.^[1]

The authors argued that the naive estimator applied a Ridge-like shrinkage and then a Lasso-like threshold, creating excessive bias in their regression experiments. They defined their corrected estimate as

$$\hat{\beta}_{\mathrm{EN}} =(1+\lambda_2)\hat{\beta}_{\mathrm{naive}}.$$

That factor belongs to the paper's particular scaling. Current glmnet, scikit-learn, statsmodels, H2O, and MATLAB documentation instead describes a direct minimizer of a mixed-penalty objective. Calling those implementations "naive" can be historically informative, but it is not their public API terminology.^{[7][8][11][13][14]}

Parameter names in software

The same letters do not have the same meaning across libraries:

Scroll sideways for more →

In all rows except the original-paper convention described above, a mixing value of 1 means pure L1 and a value of 0 means pure L2. The range endpoints can still require a dedicated solver. For example, scikit-learn documents very small l1_ratio values as unreliable unless the user supplies a suitable sequence of penalty strengths.^[8]

Statistical behavior

Sparsity and shrinkage

Lasso, introduced by Robert Tibshirani in 1996, uses an L1 penalty and can set coefficients to zero.^[2] Ridge, developed for nonorthogonal regression problems by Arthur Hoerl and Robert Kennard in 1970, uses a squared L2 penalty and generally produces a dense coefficient vector.^[3] Elastic Net combines these behaviors. Its L1 part supplies thresholding, while its L2 part adds continuous shrinkage and curvature.

The resulting coefficient estimates are biased toward zero. That bias is deliberate: reducing variance can improve prediction outside the training sample. Whether the tradeoff helps is empirical. A larger penalty is not automatically safer, and an Elastic Net fit is not automatically better than its Lasso or Ridge endpoints.^[17]

Grouping effect

The original paper used grouping effect for the tendency of highly correlated predictors to have similar coefficients and to enter or leave the model together. For centered (y), standardized predictors, and two nonzero coefficients with the same sign, the paper bounded their coefficient difference by a quantity that decreases as their sample correlation approaches 1 and increases as the L2 penalty weakens. For strong negative correlation, the sign of one predictor can be reversed before making the comparison.^[1]

The extreme case is clearer. If two standardized predictor columns are identical and the L2 component is positive, strict convexity and symmetry force their fitted coefficients to be equal. A pure Lasso fit need not be unique in that situation and can assign the shared effect in different ways.^[1]

This property is weaker than a predefined group penalty. Elastic Net does not know that a set of genes, dummy variables, or sensor channels forms a semantic group. It encourages similar treatment when the observed columns are correlated, but it does not guarantee selection of a whole pathway or block.

More predictors than observations

For a Lasso problem in general position, a unique sparse solution has no more active coefficients than the rank of the design matrix. The original Elastic Net paper summarized the practical (p>n) limitation as Lasso selecting at most (n) variables before saturation. Degenerate designs can have multiple Lasso solutions, so the statement should not be treated as an unconditional count for every solver output.^[1][18]

With a positive L2 term, Elastic Net can have more than (n) nonzero coefficients. Zou and Hastie showed this by augmenting (X) with scaled identity rows and transforming the naive Elastic Net problem into a Lasso problem on a full-rank design with (n+p) rows. The construction also underlies their path algorithm.^[1]

Comparison with Lasso and Ridge

Scroll sideways for more →

No row establishes a universal prediction ranking. Ridge can perform better when many correlated predictors carry small effects, Lasso can perform better when the signal is very sparse, and an intermediate mixture can help when both patterns matter. Model selection should be based on an evaluation design that matches the intended use.

Computation

LARS-EN

The 2005 paper proposed LARS-EN, an adaptation of least-angle regression. The underlying LARS algorithm, published in 2004, computes a piecewise-linear Lasso path through a sequence of active sets.^[4] By applying LARS to the augmented design and then rescaling, LARS-EN produced the original Elastic Net path with computational effort comparable to one ordinary least-squares fit in the paper's analysis.^[1]

The augmentation can be unattractive when (p) is very large because it introduces (p) additional rows. The original paper noted this problem and developed sparse updates to avoid storing the full augmented matrix. Contemporary libraries more often use coordinate descent for this objective.

Coordinate descent

For squared error with standardized columns satisfying (n^{-1}\sum_i x_{ij}^2=1), a coordinate update has the form

$$z_j= \frac{1}{n}\sum_{i=1}^{n} x_{ij} \left( y_i-\beta_0-\sum_{k\neq j}x_{ik}\beta_k \right),$$ $$\beta_j\leftarrow \frac{S(z_j,\lambda\alpha)} {1+\lambda(1-\alpha)}, \qquad S(z,\gamma)=\operatorname{sign}(z)(|z|-\gamma)_+.$$

The soft-thresholding numerator creates zeros; the denominator supplies the additional L2 shrinkage. If columns are not on this scale, the denominator contains their own squared norms.

Friedman, Hastie, and Tibshirani described fast cyclic coordinate-descent algorithms for Gaussian, binomial, and multinomial models in 2010. Their pathwise strategy uses warm starts, so the solution at one penalty strength initializes the next.^[5] Later strong rules screen predictors that are unlikely to be active, with optimality checks used to catch screening violations.^[6] Current glmnet also uses active-set convergence and sparse-matrix support.^[7]

The Gaussian update is not copied unchanged into every model family. For logistic and other likelihoods, implementations can use an outer quadratic or Newton approximation and solve a weighted penalized problem inside it. Solver names, stopping criteria, screening rules, and support for sparse matrices differ among packages.

Tuning and preprocessing

Choosing the penalty

Both overall strength and mixture can affect prediction and the selected set. K-fold cross-validation is a common choice, but package behavior differs:

• cv.glmnet selects lambda for one fixed alpha per call. Its vignette recommends comparing separate calls with the same fold assignments when tuning alpha.^[7]
• ElasticNetCV in scikit-learn can accept multiple l1_ratio values and evaluates a path of alpha values for each one. It refits the selected configuration on the full input data.^[9]
• H2O recommends a grid over alpha with a lambda search for each mixture.^[13]

In glmnet, lambda.min minimizes mean cross-validated error. lambda.1se is the largest lambda whose estimated error is within one standard error of the minimum, so it usually gives a more regularized model.^[7] The rule is a parsimony convention, not proof that the selected variables are true or that the model will lose negligible accuracy in every new population.

Preprocessing, feature screening, and hyperparameter selection must be learned within each training fold. If the same cross-validation results are used both to tune the model and to claim final performance, an outer validation loop or untouched test set is needed for a less biased performance estimate.

Scaling and intercepts

L1 and L2 penalties are not invariant to predictor units. Without adjustment, a one-unit coefficient for a feature measured in meters receives the same formal penalty as a one-unit coefficient for a feature measured in millimeters, although the two coefficients represent different changes in the response. Centering and scaling predictors often makes a shared penalty more meaningful.^[10]

Standardization is a modeling choice rather than an absolute rule. Binary indicators, count features, sparse inputs, and variables with intentionally different penalty factors can require other treatment. The fitted transformation must also be applied consistently to new data.

glmnet, H2O GLM, Spark ML linear regression, and MATLAB lasso standardize numeric predictors by default and return coefficients on the original scale.^{[7][12][13][14]} Scikit-learn's ElasticNet does not standardize automatically; a transformer such as StandardScaler must be included explicitly when scaling is wanted.^[8][10] glmnet also standardizes a Gaussian response when constructing its default penalty path and then unstandardizes the coefficients, which matters when reproducing a fit in another package.^[7]

An intercept is usually estimated without a penalty. If data are already centered and an intercept is disabled, that choice must match both training and prediction. Penalizing a constant column by accident changes the target model.

Software implementations

The statistical method is distinct from any one implementation. Defaults and objectives must be checked before comparing coefficients.

Scroll sideways for more →

Two packages can therefore return different numbers from superficially similar settings. Common causes include loss normalization, feature and response scaling, intercept handling, observation weights, penalty grids, convergence tolerances, and whether a reported coefficient was transformed back to the original units.

Extensions and applications

The penalty can be added to a loss function other than squared error. glmnet supports binomial, multinomial, Poisson, Cox, multiresponse Gaussian, and custom generalized linear model families.^[7] A penalized logistic regression still models a log-odds likelihood; Elastic Net describes its coefficient penalty, not a replacement probability model.

The adaptive Elastic Net assigns predictor-specific weights to the L1 terms. Hui Zou and Hao Helen Zhang established an oracle property under their stated high-dimensional regularity conditions. The result is asymptotic and assumption-dependent, not a guarantee that an adaptive fit recovers the true variables in a finite sample.^[15]

Multi-task Elastic Net is also a different objective. Scikit-learn penalizes the row-wise (L_{2,1}) norm of a coefficient matrix together with its squared Frobenius norm. This encourages a predictor to be active across several response tasks, rather than fitting each response with an independent coefficientwise L1 penalty.^[16]

The original paper used microarray data as a high-dimensional demonstration. Its leukemia data contained 7,129 genes and 72 samples, split into 38 training and 34 test samples. Within each training fold, the authors first screened to 1,000 genes. Their selected Elastic Net model used 45 genes, had 3 errors among the 38 cross-validation predictions, and made no errors on the fixed 34-sample test set.^[1] This was one historical experiment with a small test set, not evidence that Elastic Net is universally the best classifier or gene-selection method.

More generally, Elastic Net is considered when predictors are numerous, correlated, and intended for a linear or generalized-linear prediction model. Text features, molecular measurements, economic indicators, and sensor channels can have that structure, but subject area alone does not justify the method. The loss, sampling design, missing-data process, dependence structure, and validation plan still determine whether the model is suitable.

Limitations and interpretation

• Selection is tuning-dependent. Small changes in folds, scaling, or mixture can change which coefficients are zero.
• Grouping is not semantic grouping. Correlation can arise from redundancy, measurement construction, confounding, or leakage. Similar coefficients do not prove a shared mechanism.
• Coefficients are biased. Shrinkage improves some prediction problems by accepting bias. Raw penalized coefficients are not unbiased effect estimates.
• Ordinary inference does not survive selection automatically. Standard least-squares confidence intervals and p-values do not account for choosing variables and penalties on the same data. Valid post-selection procedures condition on or otherwise account for that selection step.^[19]
• The model remains linear in its encoded features. Nonlinearity and interactions must be represented explicitly or handled by a different model.
• No universal dominance exists. The original paper reported favorable experiments, but a new application must compare candidate models on appropriate held-out data.

History

Zou and Hastie's manuscript was dated December 5, 2003 and revised in August 2004. The journal article was published online on March 9, 2005 in volume 67 of the Journal of the Royal Statistical Society, Series B.^[1] It built on the earlier shrinkage traditions of Ridge and Lasso and on the LARS path algorithm.^[2][3][4]

The 2010 coordinate-descent paper supplied algorithms that became central to glmnet and broadened efficient path fitting beyond Gaussian regression.^[5] Strong screening rules followed in 2012.^[6] Later libraries adopted the same general penalty with their own parameter names, scaling conventions, families, and solvers. As a result, a reproducible report should name the software and version, objective or family, preprocessing, penalty grid, cross-validation splits, stopping tolerance, and final selected parameters.

See also

• L1 regularization
• L2 regularization

References

1. ^Hui Zou and Trevor Hastie, "Regularization and Variable Selection via the Elastic Net," *Journal of the Royal Statistical Society: Series B*, 67(2), 301-320, 2005. doi.org/...j.1467-9868.2005.00503.x
2. ^Robert Tibshirani, "Regression Shrinkage and Selection via the Lasso," *Journal of the Royal Statistical Society: Series B*, 58(1), 267-288, 1996. doi.org/...j.2517-6161.1996.tb02080.x
3. ^Arthur E. Hoerl and Robert W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems," *Technometrics*, 12(1), 55-67, 1970. doi.org/...00401706.1970.10488634
4. ^Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani, "Least Angle Regression," *The Annals of Statistics*, 32(2), 407-499, 2004. arxiv.org/...0406456
5. ^Jerome Friedman, Trevor Hastie, and Robert Tibshirani, "Regularization Paths for Generalized Linear Models via Coordinate Descent," *Journal of Statistical Software*, 33(1), 1-22, 2010. jstatsoft.org/...v033i01
6. ^Robert Tibshirani, Jacob Bien, Jerome Friedman, et al., "Strong Rules for Discarding Predictors in Lasso-type Problems," *Journal of the Royal Statistical Society: Series B*, 74(2), 245-266, 2012. pmc.ncbi.nlm.nih.gov/...PMC4262615
7. ^Trevor Hastie, Junyang Qian, and Kenneth Tay, "An Introduction to glmnet," April 30, 2026, accessed July 28, 2026. stat.ethz.ch/...glmnet.pdf
8. ^scikit-learn developers, "ElasticNet," scikit-learn API documentation, accessed July 28, 2026. scikit-learn.org/...sklearn.linear_model.ElasticNet
9. ^scikit-learn developers, "ElasticNetCV," scikit-learn API documentation, accessed July 28, 2026. scikit-learn.org/...earn.linear_model.ElasticNetCV
10. ^scikit-learn developers, "StandardScaler," scikit-learn API documentation, accessed July 28, 2026. scikit-learn.org/...n.preprocessing.StandardScaler
11. ^statsmodels developers, "OLS.fit_regularized," statsmodels 0.14.6 documentation, updated December 5, 2025. statsmodels.org/...inear_model.OLS.fit_regularized
12. ^Apache Spark project, "pyspark.ml.regression.LinearRegression," Spark 4.1.2 API documentation, accessed July 28, 2026. spark.apache.org/...ml.regression.LinearRegression
13. ^H2O.ai, "Generalized Linear Model," H2O 3.46.0.11 documentation, accessed July 28, 2026. docs.h2o.ai/...glm
14. ^MathWorks, "Lasso and Elastic Net," MATLAB Statistics and Machine Learning Toolbox documentation, accessed July 28, 2026. mathworks.com/...lasso-and-elastic-net
15. ^Hui Zou and Hao Helen Zhang, "On the Adaptive Elastic-Net with a Diverging Number of Parameters," *The Annals of Statistics*, 37(4), 1733-1751, 2009. doi.org/...08-AOS625
16. ^scikit-learn developers, "MultiTaskElasticNet," scikit-learn API documentation, accessed July 28, 2026. scikit-learn.org/...near_model.MultiTaskElasticNet
17. ^Jake Lever, Martin Krzywinski, and Naomi Altman, "Regularization," *Nature Methods*, 13, 803-804, 2016. doi.org/...nmeth.4014
18. ^Ryan J. Tibshirani, "The Lasso Problem and Uniqueness," *Electronic Journal of Statistics*, 7, 1456-1490, 2013. doi.org/...13-EJS815
19. ^Jason D. Lee, Dennis L. Sun, Yuekai Sun, and Jonathan E. Taylor, "Exact Post-Selection Inference, with Application to the Lasso," *The Annals of Statistics*, 44(3), 907-927, 2016. doi.org/...15-AOS1371