# Neural Networks Are Already Clean

> Source: https://aiwiki.ai/wiki/neural_networks_are_already_clean
> Updated: 2026-10-27
> Fact-checked: 2026-10-27
> Categories: AI Research, AI Safety, Interpretability, Neural Networks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Neural Networks Are Already Clean." aiwiki.ai, 27 Oct 2026. https://aiwiki.ai/wiki/neural_networks_are_already_clean
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Neural Networks Are Already Clean** is a long interactive research essay by [Nick Cammarata](https://aiwiki.ai/wiki/nick_cammarata), published by the interpretability company [Goodfire](https://aiwiki.ai/wiki/goodfire_ai) on 15 October 2026 at already-clean.goodfire.com.[1][11] Its page description calls it "a working hypothesis about neurons, generalization, and the ecology of circuits." The essay argues that the best-known evidence for [polysemanticity](https://aiwiki.ai/wiki/polysemanticity) and [superposition](https://aiwiki.ai/wiki/superposition) in vision models was misread. On Cammarata's account, most of the apparent tangling came from weak tools, weak architectures, and a pattern he calls "recursive generalization," not from networks packing unrelated concepts into the same neurons. He concludes that individual neurons and circuits can be studied directly, and that [mechanistic interpretability](https://aiwiki.ai/wiki/mechanistic_interpretability) "may be far better than we thought."[1]

The essay matters partly because of who wrote it. Cammarata co-authored "Zoom In: An Introduction to Circuits" (2020), which introduced the "cars in superposition" circuit, and he worked on the 2023 OpenAI project that used [GPT-4](https://aiwiki.ai/wiki/gpt_4) to explain neurons in [GPT-2](https://aiwiki.ai/wiki/gpt_2).[4][3] In the essay he revisits both pieces of work and argues that his earlier conclusions were wrong.[1] The claims about language models are preliminary: the essay says a technical companion paper will give the detailed language-model results, and that paper was described as forthcoming when the essay came out.[1][16] Reactions were split. Some researchers called the essay a historic result. Others, including [Google DeepMind](https://aiwiki.ai/wiki/google_deepmind) interpretability lead Neel Nanda, said it does not change the big picture and that superposition is still real.[12][13]

## Publication details

| Item | Detail |
|---|---|
| Title | Neural Networks Are Already Clean |
| Author | Nick Cammarata (byline "Nick Cammarata · Goodfire") |
| Publisher | Goodfire (listed as "Fundamental Research" on its research page) |
| Date | 15 October 2026 |
| Format | Interactive web essay with explorable "microscopes," neuron galleries and attribution-graph viewers, plus a plain-text Markdown version for "reading tools and AI agents" |
| Language-model collaborator | Curt Tigges (the essay says the language-model section "was done with Curt Tigges") |
| Models studied | InceptionV1 (original and retrained), three ConvNeXt models trained by the author, DINOv2, GPT-2 XL, Qwen3-32B, Qwen3.6-35B-A3B, Kimi K3 |
| Companion work | A "technical companion paper" on language-model neurons, described as forthcoming |
| Suggested citation | Cammarata, N., 2026. Neural Networks Are Already Clean. Goodfire. |

Sources: [1][11]

Cammarata announced the essay on X on the same day. His post said he had "started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had," and that "with modern agentic tools and a much closer look, the picture looks totally different."[2] He ended it with: "it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works." As of 27 October 2026 the post had more than 4.5 million views, according to the fxtwitter API.[2] Replying the same day to a user who asked when "the accompanying technical paper" would come out, Cammarata wrote "maybe a few weeks?"[16]

The acknowledgments single out "my coworker Curt Tigges, who did much of the language model work with me," and Tarun Gogineni for feedback on drafts, and also thank Tom McGrath and Eric Ho, among others. They list feedback from Joshua Batson (whom the essay identifies as being at [Anthropic](https://aiwiki.ai/wiki/anthropic)) and Jack Lindsey, and thank [Chris Olah](https://aiwiki.ai/wiki/chris_olah) "for years of mentorship and for the many ideas this essay builds on, even where it disagrees."[1]

## Background

### Polysemanticity and superposition

A neuron is called polysemantic when it seems to respond to several unrelated things. The standard example is unit 4e:55 of InceptionV1 ([GoogLeNet](https://aiwiki.ai/wiki/googlenet)). "Zoom In" (Olah, Cammarata, Schubert, Goh, Petrov and Carter, *Distill*, 2020) described it as "a polysemantic neuron which responds to cat faces, fronts of cars, and cat legs," and added: "To be clear, this neuron isn't responding to some commonality of cars and cat faces."[4] The same article asked why such neurons form. Its answer was that they "seem to result from a phenomenon we call 'superposition' where a circuit spreads a feature across many neurons, presumably to pack more features into the limited number of neurons it has available." It gave a single worked example of this, "Circuit 3: Cars in Superposition," in which a car feature in layer mixed4c appeared to spread into dog-detecting neurons in the next layer.[4]

Anthropic's "[Toy Models of Superposition](https://aiwiki.ai/wiki/toy_models_of_superposition)" (Elhage et al., 14 September 2022) gave the term its now-standard technical meaning. It used small ReLU networks trained on synthetic sparse features "to investigate how and when models represent more features than they have dimensions. We call this phenomenon superposition."[5] Anthropic's 2022 "Softmax Linear Units" paper stated that "superposition is often treated as the default hypothesis simply because it seems intuitively more compelling than other explanations, but there is little evidence." It then described its own results as "moderate evidence" for superposition.[8]

### Dictionary learning as the response

If neurons are knotted, the natural response is to study a different basis. "[Towards Monosemanticity](https://aiwiki.ai/wiki/towards_monosemanticity)" (Bricken et al., 2023) trained a [sparse autoencoder](https://aiwiki.ai/wiki/sparse_autoencoder) on a one-layer transformer. In its human rating study, "The median neuron scored 0 on our rubric, indicating that our annotator could not even form a hypothesis of what the neuron could represent," while the median feature interval scored 12.[6] As a motivating example, the paper also cited a neuron that "responds to a mix of academic citations, English dialogue, HTTP requests, and Korean text."[6] SAEs, transcoders and [attribution graphs](https://aiwiki.ai/wiki/attribution_graphs) built on them became a dominant research program (see [dictionary learning](https://aiwiki.ai/wiki/dictionary_learning)).

### OpenAI's automated neuron explanations (2023)

"Language models can explain neurons in language models" (Bills, Cammarata, Mossing, Tillman, Gao, Goh, Sutskever, Leike, Wu and Saunders; OpenAI, 9 May 2023) had GPT-4 write natural-language explanations of every MLP neuron in GPT-2 XL. A simulator then scored each explanation by how well it predicted the neuron's activations.[3] The contributions statement says "Nick effectively started the project by having the initial idea to have GPT-4 explain neurons."[3] Steven Bills is the first-listed author, and the paper marks Cammarata as one of several core research contributors. The paper reported that "out of a total of 307,200 neurons, 5,203 (1.7%) have top-and-random scores above 0.7 (explaining roughly half the variance)," and that it "found over 1,000 neurons with explanations that scored at least 0.8."[3] The new essay cites the 1.7% figure as the share of neurons that project found interpretable "by our most generous measure." Cammarata's X post rounds it to "about 2%."[1][2]

## Central argument

The essay opens with a two-sentence thesis: "The cause of polysemanticity in early model organisms wasn't superposition, but several ordinary, mostly fixable issues. We can now understand most neurons in models up to trillions of parameters, and think the outlook for mechanistic interpretability may be far better than we thought."[1] It sets out the argument in four numbered steps:[1]

| Step | Claim as stated in the essay |
|---|---|
| 1 | Networks use one learning principle (generalization), not two, "so there's no blocker to beginning ordinary science on them" |
| 2 | With modern models and tools, that science "goes much better than most people expect" |
| 3 | Understanding compounds, so it gets easier as the map fills in; "at minimum," a network's own mechanisms should be part of the toolkit |
| 4 | "Highly speculative": this might be most of what is needed, and interpretability "might be far more tractable, possibly in the near term, than we've assumed" |

Cammarata contrasts two approaches. One is "rapid normal science on the network itself," in which researchers study neurons and circuits and test explanations "at agentic scale." The other starts from hypothesized features and trains a decomposition such as an SAE to extract them. In his telling, the field began with the first and fell back to the second because networks seemed to be "tangled knots."[1] Borrowing [Rich Sutton](https://aiwiki.ai/wiki/richard_sutton)'s phrase, he proposes "a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition" (see [The Bitter Lesson](https://aiwiki.ai/wiki/bitter_lesson)).[1]

### Which "superposition" the essay disputes

The essay says the word "superposition" is used in several senses and narrows its target to one. That sense is a learning principle in which a network, short of room, "stops grouping related things together and instead begins folding unrelated things together, simply because they don't interfere."[1] It does not dispute that a network can represent more features than it has dimensions. Cammarata writes that he "basically" doesn't think that Toy Models sense is bad news, since "even a fully monosemantic network would satisfy" it because its neurons compose.[1] It cites Olah's 2023 informal note "Distributed Representations: Composition & Superposition," which frames composition and superposition as competing strategies, as the framing "closest to the question this essay cares about."[1] A footnote says that none of the agents the authors tried could read past the start of that note: "The note opens with a large figure embedded directly in the page as base64 image data. Many agents' web readers hit their length limit inside it, before the argument begins."[1]

The essay proposes five possible causes of apparent polysemanticity: bad tools, bad explanations built with those tools, bad learning (architecture or data), recursive generalization, and superposition. Cammarata writes that he "was making superposition the null hypothesis when it should have been the last hypothesis, since it's the most radical and the least testable."[1]

## Re-examined examples

### The cat and car neuron (InceptionV1 4e:55)

Cammarata writes that after months of seeing clean neurons, he reopened the familiar image grid for 4e:55 and asked what single thing it might detect: "it's not a cat and car neuron, it's a V-shape neuron."[1] As evidence he cites its circuit connections to "V-shaped flames, V-shaped water jets from a fireboat, V-shaped fox faces," and its negative pole, "an O, round where the V is pointed."[1] He argues that feature visualization showed whiskers because the optimization, once it found a route through cat circuitry, "adds everything that route brings."[1]

He notes that the 2021 "Multimodal Neurons" article, a study of [CLIP](https://aiwiki.ai/wiki/clip) that he co-authored, had come close to this reading. That article considered whether the neuron responded to a shared feature ("perhaps cat eyes and car lights look similar") before concluding it was "genuinely OR(cat, car)."[1][7]

### The "cars in superposition" circuit

The essay says Cammarata found the circuit that "Zoom In" presented as its superposition example in 2019, while building his first circuits interface. He then took it to Chris Olah, and "we came to the conclusion it must be doing a kind of folding, and called it superposition."[1] He now gives two reasons that reading was wrong:[1]

- **Generalized pose neurons.** Cars and dogs were grouped "because of shared pose." The essay says InceptionV1 has "more than a hundred" such pose neurons, each a clean detector of a pose shared across objects. It reports similar generalized pose neurons in [DINOv2](https://aiwiki.ai/wiki/dinov2) and in the author's ConvNeXt models.
- **Feature-visualization reward hacking.** In deeper layers, gradient-based optimization "takes the easiest path, which in InceptionV1 often means dogs." The essay says biasing the optimization toward the paths taken by top dataset images, "often only for the first 64 steps," showed that two of the four "dog" neurons were car neurons and two shared a pose between cars and dogs.

"To my knowledge, no one has published a second circuit with the right shape," the essay says.[1]

### InceptionV1's last layer

InceptionV1's final layer, mixed5b, had been treated as nearly uninterpretable. The essay quotes Liv Gorton's August 2024 assessment that "there are essentially no interpretable neurons on InceptionV1's final layer."[1][9] Cammarata gives these estimates:[1]

| Model and tools | Share of last-layer neurons the essay calls interpretable |
|---|---|
| Original 2014 InceptionV1, tools of that era | about 5% (Cammarata's recollection; "others at ~0%") |
| Original InceptionV1, improved tools (fixed feature visualization, attribution visualization) | about 50% |
| Same architecture retrained with "more reasonable modern regularization" | about 70% |
| ConvNeXt of the same size (7.06M parameters) trained on the same data | about 99% |

The essay's summary claim is that with residual connections, [LayerNorm](https://aiwiki.ai/wiki/layer_normalization) and "a couple other improvements," and "no change to model size or training data, 99-100% of neurons in that last layer are easily interpretable."[1] It attributes the improvement to architecture changes "designed to help deep layers learn better," which had "the unintended effect of making them more interpretable." The [residual connection](https://aiwiki.ai/wiki/residual_connection) is its main example.[1]

### The Towards Monosemanticity neuron

For the neuron that "Towards Monosemanticity" described as firing on citations, dialogue, HTTP requests and Korean text, the essay proposes that it may be a "don't say a dot" neuron. Its stated evidence is that nearly all the citation, dialogue and HTTP examples fire on or near a dot, and that "eight of its ten strongest negative logit connections begin with one." The Korean examples fire partway through unfinished syllables. Because the original model is not available, the essay presents this as provisional.[1]

## Recursive generalization

The essay's main alternative to superposition is "recursive generalization." It defines this as "a neuron generalizing along more than one axis, often first along one axis, then sprouting outwards in ways that are easy to follow locally, but stop it from being 'monosemantic' in the end."[1] Its headline example is a language-model neuron that responds to "iy," then to "diy," then to "do it yourself," "which no longer contains 'iy' at all."[1] Another is a planetary-geology neuron that also fires on code using the Cesium JavaScript library and the IDL language.[1]

The essay credits a subsection of "Multimodal Neurons" (2021) with describing the same pattern first. That article described "conjoined" neurons as concepts "overlapping in a superficial way in one facet, and then generalizing out in multiple directions."[1][7] Cammarata says he had forgotten the subsection until late in the project. He chose a new name because "conjoined" suggests two things joined at one point, whereas the growth he observes has several levels.[1]

The essay argues that recursive generalization is easy to test because you can watch it happen. When it retrained InceptionV1 and followed a neuron through training, it "started on yellow flowers, in photos that sometimes included a dog," then grew toward other nature, green objects such as balls, and dogs without flowers. A ConvNeXt neuron trained on the same data showed "the same pattern, but much less of it."[1] The essay contrasts the two explanations' predictions: "generalization should be local, superposition should be non-local."[1]

## Vision-model experiments

Most of the controlled experiments use vision models, which the essay says "we can see ... so clearly." Cammarata trained three ConvNeXt models "spanning roughly 300x in size": a small one with 92k parameters, a medium one with 7.06M (the size of InceptionV1), and a large one with 28.6M.[1] The essay reports these experiments:[1]

| Experiment | Reported result |
|---|---|
| InceptionV1 layer 3b cut from 480 neurons to 10 and retrained, with no bypass | Neurons stayed interpretable but coarser ("one 'curves & loops' neuron instead of ten curve detectors"); 58.35% top-1 and 81.14% top-5 ImageNet accuracy. An SAE trained on the crushed layer did not recover clean curve detectors |
| Medium ConvNeXt final layer cut from 1,024 neurons to 13, trained from scratch | Same broadening pattern, not overlapping mixtures |
| SAEs trained on ConvNeXt layers | On the last layer, "88% of its activity is near-copies of single neurons"; removing dictionary directions that match no neuron "leaves accuracy unchanged" |
| Teaching the 7.06M model that an umbrella photo is a bee | In nine steps the top guess went from umbrella to maypole to bee, via existing bee neurons in the last three blocks, not a new hidden feature |
| Residual stream narrowed four and eight times (to 46 dimensions in the last block) | Accuracy fell from 72% to 60% and 39%; the stream's map looked "a bit messier" but not "cleverly interleaved" |
| Weak activations removed in InceptionV1's final layer | Keeping roughly the strongest quarter of positive activations gave 69.47% accuracy, versus 70.08% originally |
| Adversarial examples traced through ConvNeXt and InceptionV1 | Attacks recruited recognizable parts of the target class's existing circuit, such as jaguar neurons |

The essay also revisits Gorton's "The Missing Curve Detectors of InceptionV1," which applied SAEs to InceptionV1's early layers.[18] It reports that the SAE curves it inspected "decompose into existing curve-responsive neurons, including the original ten curve neurons."[1] Cammarata was first author of the 2020 *Distill* article "Curve Detectors," which described those neurons.[10]

## Language-model findings

The essay says this section draws on "our forthcoming technical paper studying language neurons in depth," written with Curt Tigges.[1]

### Headline estimates

| Model | Estimate in the essay |
|---|---|
| Qwen3-32B (preliminary audits) | "roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood" |
| Qwen3-32B (smaller samples, strong activations only) | "closer to 97% non-arbitrary mixtures, but these are relatively small samples" |
| GPT-2 XL (500 "neuron papers" made "with almost no iteration") | about 70% "look like roughly one thing"; about 5% "pristine"; the rest do "a few broad things, up to about six, mostly two" |
| Dense and mixture-of-experts models | "seem roughly similar" |
| Kimi K3 and other models "up to trillions of parameters" | "similar results," to be covered in more detail in the companion paper |

Sources: [1]

The essay attaches a long footnote to the Qwen3-32B numbers. It says the share "varies depending on how you set the threshold for counting weak activations, depth, density of sub-circuits, and what you view as 'one thing'." It also says that "early reviewers felt unanchored without even a rough number," and that any single "cleanliness" score "should be taken as a broad pointer to the regime a neural network is in, rather than something precise."[1] The figures count neuron "poles," meaning the positive and negative sides of a neuron are assessed separately, rather than whole neurons.[1] For GPT-2 XL, the essay notes that "the scoring isn't identical to the original project."[1]

### Other language-model observations

- **Attribution graphs from neurons.** The essay rebuilds a medical-diagnosis attribution graph from Anthropic's "On the Biology of a Large Language Model," which studied [Claude 3.5 Haiku](https://aiwiki.ai/wiki/claude_3_5_haiku) with cross-layer [transcoder](https://aiwiki.ai/wiki/transcoder) features, using raw neurons in Qwen3-32B. It says the diagnosis hub is "just four neurons." It also presents what it calls, "as far as we know," the first published attribution graph of a multi-trillion-parameter model: a neuron-only graph from [Kimi K3](https://aiwiki.ai/wiki/kimi_k3) answering a question about a found wallet.[1]
- **Negative poles.** In a Santa graph, removing eight neurons whose negative side encodes Santa lore dropped "Santa" from near-certain to 6%, while removing twenty neurons labeled Christmas or Santa dropped it only to 76%.[1]
- **The "summary motif."** A Qwen neuron that standard top-k automated labeling called a "polysemantic comma neuron" turned out to track Colorado. Steering it "made the model want to go hiking," and the commas were where the model stored a running summary of context.[1]
- **The SwiGLU double negative.** In [SwiGLU](https://aiwiki.ai/wiki/swiglu) layers, a negative gate times a negative value gives a positive output. The essay says this "echo" is usually small, but "for about 4% of neurons, it's the whole top of the list." Ablating these regimes across the network "ups most benchmarks, but lowers some reasoning ones." The essay asks architecture designers to look into fixing it.[1]
- **Experts.** In [mixture-of-experts](https://aiwiki.ai/wiki/mixture_of_experts) models, "experts are far more specialized than previously reported." The essay calls MoE "an anti-superposition architecture," because a router built from "one small linear layer" would struggle to route thousands of unrelated packed concepts to one expert. It includes a map of all 10,240 experts in [Qwen3.6](https://aiwiki.ai/wiki/qwen3_6).[1]
- **Attention.** The essay says "we've made less progress on attention than on neurons." It estimates that global names for attention heads would add about 5% of "explanatory residual," a figure it says an agent reviewing 100 attribution graphs independently matched.[1]

## Methods and validation

The essay describes large-scale use of [AI agents](https://aiwiki.ai/wiki/ai_agents) as automated researchers. Cammarata writes that agents "can map out neurons and circuits, build explorable microscopes and explore them, test explanations thoroughly, and track hundreds of thousands of attribution graphs."[1] He says he started studying language-model neurons to test whether agents had research taste, and expected them to hit "intractable feature fragments in superposition," which "never came."[1] The essay publishes a gallery of 1,000 randomly chosen agent-written "neuron papers" for Qwen3-32B.[1]

Validation tests described in the essay include:[1]

| Test | Procedure | Reported result |
|---|---|---|
| Blind maximum | One agent writes an explanation; a second sees only the explanation and writes an input meant to maximally activate the neuron | The written input "can activate the neuron more strongly than any example we found in 10 billion tokens" |
| Bundled blind maximum | Cluster upstream attribution into bundles and run a blind maximum for each, with the others ablated | Checks whether a neuron's facets are related; the essay says it "doesn't give full coverage" |
| Predictions and near misses | For 200 random gallery neurons, an agent wrote five examples that should drive each neuron and five near misses that should not, before measurement | For the 183 neurons whose explanations describe the positive side, new examples fired strongly 76% of the time and near misses 11%; the explanation "won for 164, tied for 15 and lost for 4" |
| Distant-concept control | The agent wrote texts for the most distant concept it could invent, as a proxy for what superposition would pack in | Those texts fired neurons strongly "about 1% of the time," and 179 of 183 neurons never fired on them |
| Attribution-graph audits | Agents reviewed 100,000 attribution graphs; Cammarata inspected thousands by hand | Reported only as "qualitative evidence"; he does not "trust our current tools enough to turn it into a cleanliness score" |

The essay doubts simulation-based scoring, the method behind the 2023 OpenAI numbers. It argues that simulation scores "often end up measuring where attention puts information" and reward "fine details of weak activations that matter little for understanding what a neuron does." It adds that "Cats and cars" could pass many tests of the V neuron "while completely missing what it really is."[1]

## Weak activations, scale and the "circuit ecology" hypothesis

On weak activations, the essay argues that "you can ablate almost all the weak activations of the network for almost no cost." It says what is lost is usually "fine distinctions between things the neural network is already confused about."[1] It shows that 100 million tokens through one Qwen3-32B neuron leave almost all activations near zero, with "about 160" reaching the strong half.[1] An appendix is more cautious: "We don't yet have a semantic test for weak activations that we trust," and spot checks "cannot establish what weak activations mean in general."[1]

To explain why networks stay organized, the essay proposes a "circuit ecology" or "circuit racing" picture. Circuits compete to capture remaining error, and those that reuse existing structure grow fastest and "starve" competitors. It links this to gradient starvation, to Arpit et al.'s 2017 finding that networks fit real patterns before noise, and to Chatterjee's "coherent gradients."[1] From this it predicts that the long tail of rare facts, often assumed to be where superposition is most likely, is "where it should be least likely." It calls this prediction testable and adds: "We're not at all sure about this."[1]

On "Toy Models of Superposition," the essay says "we found no problems with its experiments." It argues that the toy setup prices the cost of sharing a neuron (interference) but not the benefit (reuse). When it combined correlated inputs and hidden neurons in one toy model, "related inputs grew onto the neurons."[1]

## Stated limitations

The essay qualifies its own claims in several places:[1]

- In "On the Title," the authors say they considered "Neural Networks Are Cleaner Than We Thought" but rejected it because the update is "a change of kind." They add: "We're maybe 85% sure, but that means 85% sure that nearly everything comes from the second kind of process, which is very different from thinking 85% of mechanisms do."
- The title "overstates the evidence, because we haven't audited everything."
- Frontier language models "aren't close to solved due to sheer size."
- The essay has not "thoroughly shown that every case of heavy polysemanticity is recursive generalization."
- One InceptionV1 neuron (mixed5a:719), which responds to a pinched-center shape and to items that co-occur with it, is listed as an "unresolved" exception.
- Attention is "mostly outside the scope of this essay."

## Reception

The essay drew wide attention on X and in AI news coverage. The Decoder noted that Goodfire, as an interpretability company, has an interest in the thesis.[12]

**Supportive.** Roon (@tszzl), an OpenAI employee who The Decoder reports is a close friend of Cammarata's, posted that "i think this may be remembered as a major moment in scientific history," and that "we can start making alignment into a 'normal science'."[15][12] The Decoder added that Roon had some reservations, including a suspicion that interpretability breakthroughs will push AI capabilities forward, and that others in the field still needed to confirm the results.[12]

**Measured or skeptical.** Neel Nanda, who leads interpretability at Google DeepMind according to The Decoder, posted a response on X and cross-posted it to [LessWrong](https://aiwiki.ai/wiki/lesswrong).[12][13][14] He wrote: "This is interesting work, but I don't think it's a significant update to the big picture of interpretability." He supported "exploratory interpretability with neurons" for pragmatic questions, said using neurons "is just vastly easier" on frontier models, and called the SwiGLU double-negative observation "a useful point." But he did not accept "grander inferences like that this is substantial progress towards ambitious reverse engineering."[13] He added: "I still think superposition is real and this work doesn't really affect my cruxes." His reasons were that language models seem to hold "far more facts and memorized things in them than they have neurons," that he expects weak activations to hold "lots of semantic superposition-y info," and that he is "skeptical of how much vision results will transfer." He also said the post "defines monosemantic too loosely." He credited Cammarata for "updating a bunch of his historical vision work."[13]

On the LessWrong linkpost, user StefanHex wrote that the essay "makes valuable contributions" but that he was "not convinced by the big claims (at this time)."[14] He questioned the compression experiments, saying superposition is generally expected to work only "in high dimensions / for large numbers of neurons," so its absence in a 10-neuron layer is weak evidence. He also said LLMs "still seem to perform more independent computations (incl. fact lookups) than they have neurons."[14] He noted that Transluce's earlier neuron-description work (Choi et al.) had also suggested that neurons may be more monosemantic than previously thought.[14][17] Another commenter, Alex Gibson, objected to the framing of Cammarata's X post rather than the essay. He quoted its line "I now think ambitious interpretability will be solved by the time we need it most" and argued that interpretability posts that overclaim do well with lay audiences.[14] Another commenter, johnswentworth, said that the meaning of each neuron on the essay's "microscope" page was "not quickly obvious" to him, and asked for an artifact that explains every neuron in the network.[14]

The Decoder wrote that "part of the dispute comes down to definitions." Cammarata accepts that models can represent more concepts than they have dimensions, while Nanda sees him as agreeing on that point. What Cammarata disputes is that networks arbitrarily pack completely unrelated things together.[12]

## Comparison of interpretability estimates

The figures below come from different models, methods and definitions, so they are not directly comparable.

| Source | Model | Measure | Figure |
|---|---|---|---|
| Bills et al. (OpenAI, 2023)[3] | GPT-2 XL, all 307,200 MLP neurons | Simulation "top-and-random" score above 0.7 | 5,203 neurons (1.7%) |
| Bills et al. (OpenAI, 2023)[3] | GPT-2 XL | Explanation score at least 0.8 | "over 1,000 neurons" |
| Towards Monosemanticity (Anthropic, 2023)[6] | One-layer transformer | Median neuron on a 0-14 human rubric | 0 |
| Gorton (2024)[9] | InceptionV1, mixed5b | Qualitative | "essentially no interpretable neurons" |
| Already Clean (Goodfire, 2026)[1] | ConvNeXt, 7.06M parameters | Last-layer neurons "easily interpretable" | about 99% |
| Already Clean (Goodfire, 2026)[1] | GPT-2 XL, 500 neurons | "roughly one thing" | about 70% |
| Already Clean (Goodfire, 2026)[1] | Qwen3-32B, preliminary audits | Neuron poles with "one main job" | roughly 90% |

## See also

- [Mechanistic interpretability](https://aiwiki.ai/wiki/mechanistic_interpretability)
- [Superposition (mechanistic interpretability)](https://aiwiki.ai/wiki/superposition)
- [Polysemanticity](https://aiwiki.ai/wiki/polysemanticity)
- [Monosemanticity](https://aiwiki.ai/wiki/monosemanticity)
- [Sparse autoencoder](https://aiwiki.ai/wiki/sparse_autoencoder)
- [Circuit discovery](https://aiwiki.ai/wiki/circuit_discovery)
- [OpenAI Microscope](https://aiwiki.ai/wiki/microscope)

## References

1. Cammarata, N. "Neural Networks Are Already Clean." Goodfire, 15 October 2026. https://already-clean.goodfire.com/ (plain-text version: https://already-clean.goodfire.com/essay.md). Accessed 27 October 2026.
2. Cammarata, N. (@nickcammarata). Post on X announcing the essay, 15 October 2026. https://x.com/nickcammarata/status/2110757819427574056. Accessed 27 October 2026.
3. Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J. and Saunders, W. "Language models can explain neurons in language models." OpenAI, 9 May 2023. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html. Accessed 27 October 2026.
4. Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S. "Zoom In: An Introduction to Circuits." *Distill*, 2020. https://distill.pub/2020/circuits/zoom-in/. Accessed 27 October 2026.
5. Elhage, N., Hume, T., Olsson, C., et al. "Toy Models of Superposition." *Transformer Circuits Thread*, 14 September 2022. https://transformer-circuits.pub/2022/toy_model/index.html. Accessed 27 October 2026.
6. Bricken, T., Templeton, A., Batson, J., et al. "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning." *Transformer Circuits Thread*, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html. Accessed 27 October 2026.
7. Goh, G., Cammarata, N., Voss, C., et al. "Multimodal Neurons in Artificial Neural Networks." *Distill*, 2021. https://distill.pub/2021/multimodal-neurons/. Accessed 27 October 2026.
8. Elhage, N., Hume, T., Olsson, C., et al. "Softmax Linear Units." *Transformer Circuits Thread*, 2022. https://transformer-circuits.pub/2022/solu/index.html. Accessed 27 October 2026.
9. Gorton, L. "Interpretable Features and Circuits in InceptionV1's Mixed5b." Blog post, 7 August 2024. https://livgorton.com/inceptionv1-mixed5b-sparse-autoencoders/. Accessed 27 October 2026.
10. Cammarata, N., Goh, G., Carter, S., Schubert, L., Petrov, M. and Olah, C. "Curve Detectors." *Distill*, 17 June 2020. https://distill.pub/2020/circuits/curve-detectors/. Accessed 27 October 2026.
11. Goodfire. "Research." https://www.goodfire.com/research. Accessed 27 October 2026.
12. Schreiner, M. "Is AI interpretability close to a breakthrough? A new essay has researchers divided." *The Decoder*, 16 October 2026. https://the-decoder.com/is-ai-interpretability-close-to-a-breakthrough-a-new-essay-has-researchers-divided/. Accessed 27 October 2026.
13. Nanda, N. (@NeelNanda5). Post on X responding to the essay, 15 October 2026. https://x.com/NeelNanda5/status/2110875974296654062. Accessed 27 October 2026.
14. "Linkpost: Neural Networks Are Already Clean" and comments. LessWrong, October 2026. https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean. Accessed 27 October 2026.
15. Roon (@tszzl). Post on X, 15 October 2026. https://x.com/tszzl/status/2110758311339729061. Accessed 27 October 2026.
16. Cammarata, N. (@nickcammarata). Reply on X about the companion paper, 15 October 2026. https://x.com/nickcammarata/status/2110772886118228457. Accessed 27 October 2026.
17. Choi, D., Huang, V., Meng, K., Johnson, D. D., Steinhardt, J. and Schwettmann, S. "Scaling Automatic Neuron Description." Transluce, 23 October 2024. https://transluce.org/neuron-descriptions. Accessed 27 October 2026.
18. Gorton, L. "The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision." arXiv:2406.03662, 2024. https://arxiv.org/abs/2406.03662. Accessed 27 October 2026.