# Sparse Coding

> Source: https://aiwiki.ai/wiki/sparse_coding
> Updated: 2026-06-07
> Categories: Interpretability, Machine Learning, Neural Networks
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

Sparse coding is a representation learning principle in which a signal is encoded as a linear combination of a small number of elements drawn from a (usually overcomplete) dictionary of atoms. Formally, given a signal x in R^n, sparse coding seeks coefficients a in R^m and a dictionary D in R^{n x m} such that x is approximately equal to D a while the coefficient vector a has few nonzero entries. The idea was introduced as a computational model of primary visual cortex by Bruno Olshausen and David Field in 1996, who showed that maximizing sparseness of a linear code for natural images produces basis functions resembling V1 simple-cell receptive fields.[^1] In the decades since, the same mathematical template has underpinned dictionary learning, compressed sensing, sparse principal component analysis, and (most recently) the [sparse autoencoders](/wiki/sparse_autoencoder) used in [mechanistic interpretability](/wiki/mechanistic_interpretability) to decompose [large language model](/wiki/large_language_model) activations into approximately monosemantic features.[^2][^3][^4]

## History and origins

### Neuroscience motivation

The neuroscientific roots of sparse coding lie in attempts to explain why the receptive fields of simple cells in the mammalian primary visual cortex (V1) are localized, oriented, and bandpass. Earlier "efficient coding" arguments by Horace Barlow and others held that sensory neurons should reduce statistical redundancy in their inputs. Olshausen and Field proposed a stronger constraint: that the cortex should represent natural images by a sparse code, in which only a small fraction of neurons are active for any given stimulus.[^1] Their 1996 *Nature* paper introduced a generative model in which each image patch is reconstructed as a linear combination of basis functions with coefficients drawn from a sparse prior. When the basis functions were learned by gradient descent on patches sampled from photographs of natural scenes, the resulting filters formed a complete family of localized, oriented, bandpass functions reminiscent of V1 simple cells.[^1]

A 1997 follow-up in *Vision Research*, "Sparse coding with an overcomplete basis set: A strategy employed by V1?", extended the model to overcomplete dictionaries in which the number of code elements exceeds the dimensionality of the input. Olshausen and Field argued that overcompleteness allows the code to capture richer structure, such as combinations of orientation, spatial frequency, and position, and that the sparsity prior is essential for selecting a unique decomposition among the many possible ones.[^5] These two papers are commonly cited as the origin of sparse coding as a learning principle in computational neuroscience and machine learning.

### Mathematical formalization in signal processing

In parallel with the neuroscience work, the signal processing community developed a closely related theory of sparse representations over fixed dictionaries. Stephane Mallat's matching pursuit algorithm (1993) and the basis pursuit principle introduced by Scott Chen, David Donoho, and Michael Saunders in 1998 formalized the search for the sparsest decomposition of a signal over an overcomplete basis.[^6] Basis pursuit replaced the (combinatorially hard) L0 minimization by an L1 surrogate that can be solved by linear programming, and its 1998 paper "Atomic Decomposition by Basis Pursuit" in *SIAM Journal on Scientific Computing* became one of the most cited works in convex relaxation of sparsity-constrained problems.[^6]

### Dictionary learning and the K-SVD era

Olshausen and Field's algorithm jointly learned both the dictionary and the codes. In the early 2000s, Michael Elad and collaborators reformulated this idea as a general "dictionary learning" problem and developed algorithms that scale to image processing. The most influential was K-SVD, introduced by Michal Aharon, Michael Elad, and Alfred Bruckstein in 2006 in *IEEE Transactions on Signal Processing*.[^7] K-SVD generalizes the K-means clustering algorithm to overcomplete dictionaries and iteratively updates one atom at a time via singular value decomposition of the residual. Together with denoising and inpainting demonstrations by Elad and Aharon in 2006, K-SVD made sparse-representation methods a standard tool for image restoration through roughly 2012.[^8]

Online and large-scale variants followed. Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro proposed an online dictionary learning algorithm in 2009 (ICML) that uses stochastic approximation to scale to datasets with millions of patches; the journal version appeared in *Journal of Machine Learning Research* in 2010.[^9] This online formulation underlies the dictionary learning implementations in scikit-learn and SPAMS, and it is conceptually closely related to the streaming training of modern sparse autoencoders on transformer activations.

### Compressed sensing

A separate strand emerging in 2004 to 2006 connected sparse coding to sensing and acquisition. Emmanuel Candes, Justin Romberg, and Terence Tao showed that signals that are sparse in some basis can be recovered exactly from far fewer linear measurements than the Shannon sampling rate would require, provided the measurement matrix satisfies a Restricted Isometry Property. Their paper "Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information" appeared in *IEEE Transactions on Information Theory* in 2006.[^10] David Donoho's complementary article "Compressed sensing", in the same journal that year, framed the broader research program.[^11] These results gave sparse representations a recovery theory: under explicit conditions on the dictionary, the L1-minimizing solution equals the (otherwise NP-hard) L0-sparsest solution. Compressed sensing has since been applied across medical imaging (notably faster MRI), seismic acquisition, single-pixel cameras, and statistical estimation.

### Modern revival in interpretability

After 2015, sparse coding fell out of fashion in mainstream computer vision as end-to-end [deep learning](/wiki/deep_learning) surpassed dictionary-based pipelines. It returned to prominence in 2023 as a tool for interpreting trained [neural networks](/wiki/neural_network), particularly large language models. Nelson Elhage and colleagues at [Anthropic](/wiki/anthropic) proposed in "Toy Models of Superposition" (2022) that networks pack more features into their activation spaces than they have dimensions by representing them as sparse, nearly orthogonal directions, and that this "superposition" is the proximate cause of polysemantic neurons.[^12] Sparse coding then suggested a remedy: learn an overcomplete dictionary whose atoms correspond to the individual features that the network has put into superposition. Hoagy Cunningham and collaborators demonstrated in September 2023 that [sparse autoencoders](/wiki/sparse_autoencoder) trained on the residual stream of language models recover "highly interpretable features" that are more monosemantic than directions found by alternative methods.[^3] Anthropic's "[Towards Monosemanticity](/wiki/towards_monosemanticity)" (Bricken et al., October 2023) presented a detailed study of sparse autoencoders applied to a one-layer transformer, training dictionaries with expansion factors up to 256x on roughly 8 billion tokens.[^2] In May 2024, the "[Scaling Monosemanticity](/wiki/scaling_monosemanticity)" report by Adly Templeton et al. extended the approach to [Claude 3](/wiki/claude_3_opus) Sonnet, extracting more than 34 million features from a frontier production model and demonstrating "feature steering" by clamping individual dictionary coefficients.[^4]

## Mathematical formulation

### The sparse coding optimization

A standard formulation of sparse coding takes a data sample x in R^n, a dictionary D in R^{n x m} with columns called atoms, and an unknown coefficient vector a in R^m. The ideal "L0 problem" is

minimize_a ||a||_0  subject to  ||x - D a||_2 <= epsilon,

where ||a||_0 counts the number of nonzero entries of a. This problem is NP-hard in general. Practical sparse coding therefore relies on two surrogate strategies:[^7][^11]

1. **Convex relaxation** to L1. Replace ||a||_0 by ||a||_1 = sum |a_i|, giving the Lagrangian form

    minimize_a (1/2) ||x - D a||_2^2 + lambda ||a||_1,

    which is convex, often called the Lasso, and admits efficient solvers.[^6]
2. **Greedy approximation** via matching pursuit or orthogonal matching pursuit, which iteratively select the dictionary atom most correlated with the current residual.[^13]

### Algorithms for sparse coding

Three algorithm families dominate practice:

- **Iterative Shrinkage-Thresholding (ISTA).** ISTA is the canonical proximal gradient method for L1-regularized least squares. Each iteration applies a gradient step on the reconstruction loss followed by a soft-thresholding operator that promotes sparsity. ISTA converges at rate O(1/k).[^14]
- **Fast ISTA (FISTA).** Amir Beck and Marc Teboulle's accelerated variant from 2009 uses Nesterov-style momentum to attain O(1/k^2) convergence and remains a default solver for sparse coding subproblems.[^14]
- **Orthogonal Matching Pursuit (OMP).** A greedy algorithm that adds one atom at a time, then projects the signal onto the span of the selected atoms before computing the next residual.[^13] OMP is well suited when the sparsity level k is small and known.

### The dictionary learning problem

In dictionary learning, the dictionary D is itself optimized to minimize reconstruction error over a dataset X = (x_1, ..., x_N):

minimize_{D, A} sum_i (1/2) ||x_i - D a_i||_2^2 + lambda ||a_i||_1,  subject to constraints on D.

Typically the columns of D are constrained to unit L2 norm to prevent trivial rescaling. The problem is non-convex but bi-convex (convex in each of D and A with the other fixed), so most algorithms alternate:

- **Sparse coding step.** Fix D and solve for each a_i using ISTA, FISTA, or OMP.
- **Dictionary update step.** Fix A and update D.

K-SVD updates atoms one at a time, while the Method of Optimal Directions (MOD), the gradient method of Olshausen and Field, and online dictionary learning use different update rules.[^1][^7][^9] The connection to K-means is direct: when sparsity is forced to exactly one nonzero per signal, dictionary learning reduces to vector quantization.[^7]

### Recovery and identifiability

When can the sparse code be recovered uniquely? Two conditions dominate the theory:

- **Mutual coherence.** The mutual coherence mu(D) is the largest absolute inner product between distinct unit-norm atoms. If a signal has a representation with at most (1/2)(1 + 1/mu(D)) nonzeros, then this representation is unique and both basis pursuit and OMP recover it.[^13]
- **Restricted Isometry Property (RIP).** Candes and Tao showed that if D acts as an approximate isometry on all k-sparse vectors (the k-th restricted isometry constant is small), then L1 minimization recovers any k-sparse signal from y = D a, even with noise.[^10] Random matrices with Gaussian or Bernoulli entries satisfy RIP with high probability when the number of measurements scales as O(k log(n/k)).[^10][^11]

Identifiability of the dictionary itself (recovering D from sufficiently many samples up to permutation and sign of its columns) has been studied under generative assumptions on the sparse codes. Sample complexity bounds typically require the number of training samples to grow polynomially in the dictionary size and inversely in a sparsity parameter, though the constants depend strongly on the noise model.

## How sparse coding relates to neighbouring methods

| Method | Generative model | Sparsity mechanism | Inference cost |
|---|---|---|---|
| Sparse coding | x = D a + noise, a sparse | L1 or L0 penalty on a | Iterative (ISTA / OMP) |
| [PCA](/wiki/principal_component_analysis_pca) | x = D a + noise, a Gaussian | None | Single matrix multiply |
| Sparse PCA | Like PCA with sparse loadings on D | L1 penalty on columns of D | Coordinate descent or elastic net |
| Independent Component Analysis | x = D a, a independent non-Gaussian | Heavy-tailed prior or kurtosis | Linear (fixed unmixing matrix) |
| K-means / Vector quantization | x = D a, a one-hot | Cardinality constraint ||a||_0 = 1 | Nearest-neighbour search |
| [Autoencoder](/wiki/autoencoder) | x = g(f(x)) | Architectural bottleneck or L1 on f(x) | Single forward pass |

The contrast with [principal component analysis](/wiki/principal_component_analysis) is informative. PCA finds an orthonormal basis under which the data has uncorrelated coefficients of decaying variance; it is a special case of sparse coding only in the degenerate limit where the prior on coefficients is Gaussian and the dictionary is square and orthogonal. Sparse PCA, introduced by Hui Zou, Trevor Hastie, and Robert Tibshirani in 2006, reformulates PCA as a regression problem and adds an L1 (lasso) or elastic-net penalty so that each principal axis depends on only a small subset of input variables, trading off variance explained for interpretability.[^15]

Independent Component Analysis (ICA) seeks a (usually square) unmixing matrix W such that W x has statistically independent rows. Although sparse coding and ICA both arrive at oriented filters on natural images, they differ in three ways. First, ICA's forward transformation is linear (a single matrix multiply) while sparse coding's inference involves an optimization. Second, ICA typically uses square or under-complete dictionaries, whereas sparse coding is naturally overcomplete. Third, ICA does not explicitly enforce sparsity; it instead maximizes non-Gaussianity, which often produces sparse outputs as a side effect but does not give the analyst direct control over the sparsity level.[^5]

Sparse Bayesian learning, introduced by Michael Tipping in 2001, places independent zero-mean Gaussian priors on each coefficient with separate variance hyperparameters and uses evidence maximization to drive most variances to zero, yielding a sparse posterior mean. The same machinery underlies the Relevance Vector Machine, and the approach gives uncertainty estimates that hard-thresholded L1 methods do not.[^16]

## Modern application: sparse autoencoders in mechanistic interpretability

### The superposition hypothesis

The bridge from classical sparse coding to modern [mechanistic interpretability](/wiki/mechanistic_interpretability) runs through the superposition hypothesis. Toy Models of Superposition (Elhage et al., September 2022) showed that small networks trained on synthetic feature-detection tasks pack more features than they have hidden dimensions by representing them as sparse, approximately orthogonal directions, with individual neurons becoming polysemantic responders to several features at once.[^12] The paper identified a phase transition between regimes in which features are represented privately or in superposition, with the transition driven by feature sparsity and the ratio of features to dimensions, and proposed that this geometry is the structural cause of polysemantic neurons in larger language models.

If activations are sparse mixtures of feature directions, the inverse problem of recovering those directions is exactly the dictionary learning problem of Olshausen and Field. A [sparse autoencoder](/wiki/sparse_autoencoder) (SAE) implements this inversion: an encoder maps a model's hidden state h to a high-dimensional, sparsely activating code z (via a linear layer with a thresholded or top-k activation), and a decoder reconstructs h from z. The encoder weights and decoder weights are trained jointly with a reconstruction loss plus an L1 penalty (or top-k constraint) on z. In effect, the SAE learns an overcomplete dictionary D (the decoder matrix) whose atoms are interpreted as features used by the language model.[^2][^3]

### Cunningham et al. (2023) and the proof of concept

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey published the first widely cited demonstration that SAEs find interpretable features in language models. Their September 2023 arXiv paper trained sparse autoencoders on residual stream activations of small language models and showed that the resulting features were substantially more interpretable than directions found by linear probes or by examining raw neurons.[^3] The authors connected their work explicitly to sparse coding, noting that the same algorithm Olshausen and Field had used on natural images now produced semantically meaningful features when applied to transformer hidden states.

### Towards Monosemanticity (Bricken et al., 2023)

Anthropic's October 2023 report "[Towards Monosemanticity](/wiki/towards_monosemanticity): Decomposing Language Models With Dictionary Learning", led by Trenton Bricken, applied SAEs to a one-layer transformer's MLP activations, training dictionaries with expansion factors ranging from 1x to 256x on approximately 8 billion data points.[^2] The report documented thousands of features that activated on narrow, human-interpretable concepts (specific languages, syntactic constructions, named entities, DNA motifs) and showed that ablating features causally affected downstream model behaviour. It also formalized criteria for "monosemanticity" of features and showed that increasing the SAE dictionary size produced more, finer-grained features without saturating, suggesting that a large model's true feature inventory is much larger than its hidden width.[^2]

### Scaling Monosemanticity (Templeton et al., 2024)

The follow-up "[Scaling Monosemanticity](/wiki/scaling_monosemanticity): Extracting Interpretable Features from Claude 3 Sonnet" was published on the Transformer Circuits Thread in May 2024 by Adly Templeton, Tom Conerly, Jonathan Marcus, and colleagues.[^4] The team trained three sparse autoencoders on the middle-layer residual stream of [Claude 3](/wiki/claude_3_opus) Sonnet, a production model, extracting roughly 1 million, 4 million, and 34 million features respectively. Features ranged from very abstract concepts (deception, code injection vulnerabilities, sycophancy) to highly specific ones (the Golden Gate Bridge, particular cities, individual people).[^4] The authors demonstrated feature steering: clamping a feature's activation modified the model's behaviour in ways that matched the feature's interpreted meaning. This was the first demonstration that the sparse coding template scales to frontier-scale language models and provides a tractable approximate decomposition of their internal computation.

### Limitations of sparse autoencoders

Sparse autoencoders inherit several known limitations from classical sparse coding, and add new ones specific to neural-network interpretability:

- Features are sensitive to the choice of L1 penalty, expansion factor, and initialization, and different SAE runs on the same model can produce systematically different feature sets, raising questions about canonicalness.[^17]
- Reconstruction is imperfect, and the loss recovered by the SAE plus its sparsity constraint is an approximation to the model's true computation, not an exact decomposition.
- Whether features identified by SAEs are causally responsible for behaviour, or merely correlated with it, remains an open empirical question that requires intervention studies.

A 2025 ICLR paper "Sparse Autoencoders Do Not Find Canonical Units" provided evidence that the feature set discovered by an SAE depends heavily on hyperparameters and architecture choices, complicating claims that they reveal "the" features of a network.[^17]

## Applications outside interpretability

### Image processing

The deepest deployment of sparse coding has been in image restoration. The Elad and Aharon 2006 method, using a K-SVD dictionary trained on either the noisy image itself or on a corpus of clean patches, became a standard baseline for image denoising and was state of the art (alongside BM3D) for several years.[^8] Sparse representation models were extended to inpainting (filling missing pixels), demosaicing (reconstructing colour from Bayer patterns), super-resolution (recovering high-resolution detail by coupling low-resolution and high-resolution dictionaries), and compression. Many of these systems were eventually displaced by convolutional networks after roughly 2014, though sparse-coding ideas remain in the analysis of unrolled networks like LISTA.

### Classification

Sparse representations have also been used as features for classification. Sparse representation-based classification, popularized by John Wright and colleagues, represents a test sample as a sparse combination of training samples and assigns it to the class whose atoms dominate the representation. This worked notably well on face recognition tasks before CNNs took over the field.

### Audio source separation

In audio, sparse coding and non-negative sparse coding (Patrik Hoyer, 2002) have been applied to single-channel source separation. The approach assumes that source spectra can be expressed as sparse non-negative combinations of basis spectra learned for each source class. The framework generalizes Non-negative Matrix Factorization (NMF) by enforcing explicit sparsity, and underlies several music-transcription and speech-separation systems.[^18]

### Neuroscience modeling

In computational neuroscience, sparse coding remains a primary normative model of cortical representation. Beyond V1, sparse-coding networks have been used to model auditory cortex, the retina, and higher-order visual areas. The framework also provides a normative account for the empirical observation that cortical activity is sparse: a typical neuron fires only for a small fraction of stimuli, and a typical stimulus activates only a small fraction of neurons.

### Compressed sensing in acquisition

In imaging hardware, compressed sensing has been applied to accelerate magnetic resonance imaging (subsampling k-space and recovering via sparse priors in wavelet or finite-difference dictionaries), to enable single-pixel cameras that reconstruct an image from a sequence of random projections, and to seismic data acquisition.[^11]

## Limitations and criticisms

Sparse coding has several persistent weaknesses:

- **Inference cost.** Computing the sparse code for a new sample requires solving an optimization problem, which is slower than a single forward pass through an [autoencoder](/wiki/autoencoder) or a [neural network](/wiki/neural_network). Algorithms like LISTA (Learned ISTA) attempt to amortize this cost by training a small unrolled network to mimic ISTA's output, blurring the distinction between sparse coding and feed-forward inference.
- **Hyperparameter sensitivity.** Performance depends strongly on the sparsity penalty lambda, the dictionary size m, the choice of optimizer, and (for dictionary learning) initialization. The L1 penalty also shrinks coefficients, biasing the reconstruction.
- **Translational and rotational invariance.** A naive dictionary learned on small image patches will have many shifted copies of the same feature. Convolutional sparse coding addresses this by tying weights across spatial positions, but it complicates the optimization.
- **The "right" sparsity.** It is not always clear what level of sparsity to enforce or whether L0, L1, or other penalties (e.g., L0.5, top-k, JumpReLU) are most appropriate. Different choices produce qualitatively different dictionaries.
- **Identifiability is approximate.** Even in the favourable regimes where recovery guarantees apply, the dictionary is identifiable only up to permutation and sign of its atoms, and any practical estimate is noisy. For interpretability applications, this means that two SAEs trained on the same model with different seeds may produce different but equally valid feature sets.[^17]

## Relation to the manifold and superposition hypotheses

Two ideas from modern [representation learning](/wiki/representation_learning) connect closely to sparse coding. The manifold hypothesis holds that high-dimensional natural data lie on or near low-dimensional manifolds embedded in the ambient space. Sparse coding offers one way to parameterize such manifolds: a signal on a manifold of dimension k can often be written as a sparse linear combination of k atoms from a sufficiently rich dictionary, even when the ambient dimension is much larger. The Sparse Manifold Transform synthesizes sparse coding with manifold flattening to model smooth transformations in signal space as sparse interpolations between dictionary elements.

The [superposition hypothesis](/wiki/superposition), proposed by Elhage et al. (2022) for neural networks, is essentially a sparse coding model of activations: a network's hidden state h is approximately D a where D is a dictionary of "feature directions" and a is sparse and non-negative.[^12] In this view, the polysemanticity of individual neurons is the predictable consequence of decomposing a sparse code using the wrong basis (the standard basis of the activation space), and learning the right basis (via a sparse autoencoder) recovers the original features. This unifies a 1996 neuroscience model and a 2024 interpretability technique under the same mathematical template.

## Theoretical results

Several theoretical results bear directly on when sparse coding succeeds:

- **L0/L1 equivalence.** Under mutual coherence or RIP conditions, the L1 minimizer (basis pursuit) coincides with the L0 minimizer.[^10][^13] This justifies the use of [lasso-style](/wiki/lasso_regression) [L1 regularization](/wiki/l1_regularization) as a tractable surrogate for the combinatorial sparsity penalty.
- **Sample complexity.** For dictionary learning under sparse generative assumptions, the number of training samples required to recover the dictionary (up to permutation and sign) grows polynomially in the dictionary size m and inversely in a sparsity parameter.
- **Phase transitions.** Donoho and colleagues showed that compressed-sensing recovery exhibits a sharp phase transition: when the number of measurements is below a threshold (depending on sparsity and ambient dimension), L1 recovery fails almost certainly, and above it, succeeds almost certainly. The location of the transition is sometimes called the Donoho-Tanner curve.
- **Robustness.** RIP-based guarantees extend to noisy and approximately sparse settings, with the reconstruction error bounded by a multiple of the noise level plus the best k-sparse approximation error.[^10]

## Related work and see also

- [sparse autoencoder](/wiki/sparse_autoencoder): the neural-network instantiation of dictionary learning used to decompose activations.
- [towards monosemanticity](/wiki/towards_monosemanticity): Bricken et al. (2023), Anthropic's foundational SAE demonstration on a one-layer transformer.
- [scaling monosemanticity](/wiki/scaling_monosemanticity): Templeton et al. (2024), scaling SAEs to Claude 3 Sonnet.
- [mechanistic interpretability](/wiki/mechanistic_interpretability): the broader interpretability program in which sparse coding plays a central role.
- [superposition](/wiki/superposition): the polysemanticity hypothesis that motivates sparse autoencoders.
- [Principal Component Analysis](/wiki/principal_component_analysis): an orthogonal Gaussian-prior special case.
- [Manifold learning](/wiki/manifold_learning): a family of [dimensionality reduction](/wiki/dimensionality_reduction) methods related to sparse coding via the manifold hypothesis.
- [Lasso regression](/wiki/lasso_regression) and [L1 regularization](/wiki/l1_regularization): the convex relaxation underlying most practical sparse coding.
- [Unsupervised learning](/wiki/unsupervised_learning): the broader paradigm to which sparse coding belongs.
- [autoencoder](/wiki/autoencoder): the general architecture from which sparse autoencoders specialize.
- [Convex optimization](/wiki/convex_optimization) and [gradient descent](/wiki/gradient_descent): the algorithmic toolbox.

## See also

- [sparse autoencoder](/wiki/sparse_autoencoder)
- [towards monosemanticity](/wiki/towards_monosemanticity)
- [scaling monosemanticity](/wiki/scaling_monosemanticity)
- [superposition](/wiki/superposition)
- [mechanistic interpretability](/wiki/mechanistic_interpretability)
- [principal component analysis](/wiki/principal_component_analysis)
- [manifold learning](/wiki/manifold_learning)
- [dimensionality reduction](/wiki/dimensionality_reduction)
- [lasso regression](/wiki/lasso_regression)
- [unsupervised learning](/wiki/unsupervised_learning)
- [autoencoder](/wiki/autoencoder)
- [convex optimization](/wiki/convex_optimization)
- [l1 regularization](/wiki/l1_regularization)
- [interpretability](/wiki/interpretability)

## References

[^1]: Olshausen, B. A. and Field, D. J., "Emergence of simple-cell receptive field properties by learning a sparse code for natural images", Nature 381(6583): 607-609, 1996-06-13. https://www.nature.com/articles/381607a0. Accessed 2026-05-20.
[^2]: Bricken, T., Templeton, A., Batson, J., et al., "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning", Transformer Circuits Thread (Anthropic), 2023-10-04. https://transformer-circuits.pub/2023/monosemantic-features. Accessed 2026-05-20.
[^3]: Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L., "Sparse Autoencoders Find Highly Interpretable Features in Language Models", arXiv:2309.08600, 2023-09-15. https://arxiv.org/abs/2309.08600. Accessed 2026-05-20.
[^4]: Templeton, A., Conerly, T., Marcus, J., et al., "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet", Transformer Circuits Thread (Anthropic), 2024-05-21. https://transformer-circuits.pub/2024/scaling-monosemanticity/. Accessed 2026-05-20.
[^5]: Olshausen, B. A. and Field, D. J., "Sparse coding with an overcomplete basis set: A strategy employed by V1?", Vision Research 37(23): 3311-3325, 1997-12. https://www.sciencedirect.com/science/article/pii/S0042698997001697. Accessed 2026-05-20.
[^6]: Chen, S. S., Donoho, D. L., and Saunders, M. A., "Atomic Decomposition by Basis Pursuit", SIAM Journal on Scientific Computing 20(1): 33-61, 1998. https://stanford.edu/group/SOL/papers/BasisPursuit-SIGEST.pdf. Accessed 2026-05-20.
[^7]: Aharon, M., Elad, M., and Bruckstein, A., "K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation", IEEE Transactions on Signal Processing 54(11): 4311-4322, 2006-11. https://doi.org/10.1109/TSP.2006.881199. Accessed 2026-05-20.
[^8]: Elad, M. and Aharon, M., "Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries", IEEE Transactions on Image Processing 15(12): 3736-3745, 2006-12. https://pubmed.ncbi.nlm.nih.gov/17153947/. Accessed 2026-05-20.
[^9]: Mairal, J., Bach, F., Ponce, J., and Sapiro, G., "Online Learning for Matrix Factorization and Sparse Coding", arXiv:0908.0050 / Journal of Machine Learning Research 11: 19-60, 2010. https://arxiv.org/abs/0908.0050. Accessed 2026-05-20.
[^10]: Candes, E. J., Romberg, J., and Tao, T., "Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information", IEEE Transactions on Information Theory 52(2): 489-509, 2006-02. https://candes.su.domains/publications/downloads/CompressiveSampling.pdf. Accessed 2026-05-20.
[^11]: Donoho, D. L., "Compressed sensing", IEEE Transactions on Information Theory 52(4): 1289-1306, 2006-04. https://ieeexplore.ieee.org/document/1614066. Accessed 2026-05-20.
[^12]: Elhage, N., Hume, T., Olsson, C., et al., "Toy Models of Superposition", arXiv:2209.10652 / Transformer Circuits Thread, 2022-09-21. https://arxiv.org/abs/2209.10652. Accessed 2026-05-20.
[^13]: Tropp, J. A. and Gilbert, A. C., "Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit", IEEE Transactions on Information Theory 53(12): 4655-4666, 2007-12. https://users.cms.caltech.edu/~jtropp/papers/TG07-Signal-Recovery.pdf. Accessed 2026-05-20.
[^14]: Beck, A. and Teboulle, M., "A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems", SIAM Journal on Imaging Sciences 2(1): 183-202, 2009. https://epubs.siam.org/doi/10.1137/080716542. Accessed 2026-05-20.
[^15]: Zou, H., Hastie, T., and Tibshirani, R., "Sparse Principal Component Analysis", Journal of Computational and Graphical Statistics 15(2): 265-286, 2006. https://hastie.su.domains/Papers/spc_jcgs.pdf. Accessed 2026-05-20.
[^16]: Tipping, M. E., "Sparse Bayesian Learning and the Relevance Vector Machine", Journal of Machine Learning Research 1: 211-244, 2001-09. https://www.jmlr.org/papers/volume1/tipping01a/tipping01a.pdf. Accessed 2026-05-20.
[^17]: Leask, P., Bussmann, B., Pearce, M., et al., "Sparse Autoencoders Do Not Find Canonical Units", ICLR 2025 Conference Proceedings, 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/84ca3f2d9d9bfca13f69b48ea63eb4a5-Paper-Conference.pdf. Accessed 2026-05-20.
[^18]: Hoyer, P. O., "Non-negative sparse coding", Proceedings of the IEEE Workshop on Neural Networks for Signal Processing XII: 557-565, 2002. https://www.researchgate.net/publication/3966025_Non-negative_sparse_coding. Accessed 2026-05-20.