Activation patching
Activation patching is a causal intervention technique used in mechanistic interpretability to identify which internal components of a neural network are causally responsible for a specific behaviour.
Interpretability
Explore Interpretability through related topics and the articles other pages reference most.
Most referenced in this topic
Ranked by links from other AI Wiki pages.
Explore articles
Browse subtopics (19)
Articles that also belong to these categories. Counts cover all of Interpretability.
All articles (44)Machine Learning (12)Anthropic (8)Deep Learning (8)AI Safety (6)Neural Networks (6)AI Research (4)Large Language Models (4)Computer Vision (2)Developer Tools (2)Open Source AI (2)Transformer Models (2)AI Companies (1)AI Ethics (1)Algorithms (1)Generative AI (1)Mathematics (1)Model Evaluation (1)OpenAI (1)People (1)
Showing 1-44 of 44 articles
Activation steering
Activation steering is a family of inference-time techniques in mechanistic interpretability and AI safety that modify a neural network's internal activations to influence its behavior, without retraining the…
Attribution Graphs
Attribution graphs are a mechanistic interpretability technique developed by Anthropic that traces the internal "circuits" a large language model uses to turn a specific prompt into a specific output.
Christopher Olah
Christopher Olah (commonly Chris Olah) is a Canadian machine learning researcher, a co-founder of Anthropic, and the researcher most often credited with founding mechanistic interpretability
Circuit discovery
Circuit discovery is a research program in mechanistic interpretability that aims to identify sparse computational subgraphs inside trained neural networks, called circuits, that implement specific behaviors…
Crosscoder
A crosscoder is a mechanistic interpretability tool, introduced by Anthropic in October 2024, that generalizes the sparse autoencoder (SAE) and the transcoder by learning a single shared dictionary of sparse…
DeepLIFT
DeepLIFT (Deep Learning Important FeaTures) is a feature attribution method for deep neural networks introduced by Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje at Stanford University in 2017 .
Explainable AI
Explainable AI (XAI) refers to artificial intelligence systems and techniques designed so that humans can understand how and why the system reaches its decisions, predictions, or recommendations.
Feature Importances
Feature importances are numeric scores that quantify how much each input feature contributes to the predictions of a machine learning model.
Golden Gate Claude
Golden Gate Claude was a temporary, research-oriented public demonstration released by Anthropic on May 23, 2024
Goodfire AI
Goodfire AI is a San Francisco-based artificial intelligence research lab and public benefit corporation focused on mechanistic interpretability
Grad-CAM
Grad-CAM (Gradient-weighted Class Activation Mapping) is a technique for producing visual explanations from convolutional neural network (CNN) models by using the gradients of a target class flowing into the…
Induction Heads
Induction heads are a circuit pattern in Transformer language models in which a small set of attention heads, typically spread across two layers, perform an in-context "match and copy" operation that completes…
Influence functions (machine learning)
An influence function is a tool for estimating how a machine learning model's predictions would change if a single training example were removed or perturbed, without retraining the model.
Integrated Gradients
Integrated Gradients (IG) is a feature-attribution method for explainable AI that explains a neural network prediction by assigning each input feature an importance score
LIME
LIME (Local Interpretable Model-Agnostic Explanations) is a technique for explaining individual predictions of any black-box machine learning classifier or regressor by approximating the model locally with a…
Latent Space
A latent space is the vector space a machine learning model maps its inputs into, where each input becomes a point (a latent vector or latent code) and the geometry of the space carries information the raw…
Layer-wise Relevance Propagation (LRP)
Layer-wise Relevance Propagation (LRP) is an explainable AI method that explains the prediction of a deep neural network by propagating the model's output backward through the network, layer by layer, and…
Linear Probes
A linear probe is a small linear classifier (or linear regressor) trained on the frozen internal activations of a neural network to test whether a particular concept, property, or label is linearly decodable…
Logit lens
The logit lens is a foundational technique in mechanistic interpretability for inspecting the intermediate computations of transformer language models.
Mechanistic interpretability
Mechanistic interpretability (often abbreviated as mech interp or MI) is the field that reverse-engineers the internal computations of neural networks, particularly transformers
Monosemanticity
Monosemanticity is a property of an internal feature or neuron in a neural network when that unit responds to a single, human-interpretable concept rather than to a heterogeneous collection of unrelated inputs.
On the Biology of a Large Language Model
On the Biology of a Large Language Model is a mechanistic interpretability paper published by Anthropic on March 27, 2025, in the Transformer Circuits Thread.
OpenAI Microscope
OpenAI Microscope is a publicly accessible collection of visualizations of the neurons, channels, and features inside a number of significant, commonly studied computer vision neural networks.
Patchscopes
Patchscopes is an interpretability framework for inspecting hidden representations of large language models by patching an internal activation from a source computation into a separate target inference whose…
Permutation variable importances
Permutation variable importance is a model-agnostic technique that measures how much a fitted machine learning model relies on a given feature by randomly shuffling that feature's values and recording how much…
Persona vectors
Persona vectors are single linear directions in the activation space of a large language model that correspond to high level character traits such as evil, sycophancy, or a propensity to hallucinate.
Polysemanticity
Polysemanticity is the phenomenon in artificial neural networks in which a single neuron (or directional unit such as an attention head) activates strongly for multiple, semantically unrelated inputs or…
Refusal direction
The refusal direction is a finding from mechanistic interpretability research that the refusal behavior of safety fine-tuned chat language models is mediated by a single
Representation Engineering
Representation Engineering (often abbreviated RepE) is a top-down approach to artificial-intelligence transparency and control that reads and manipulates high-level concepts (such as honesty, harmlessness, and…
SHAP (SHapley Additive exPlanations)
SHAP (SHapley Additive exPlanations) is a game-theoretic method that explains an individual machine learning prediction by assigning each input feature a numerical value representing how much it pushed that…
Saliency map
A saliency map is an explainable AI visualization that highlights which parts of an input, most often the individual pixels of an image, most influenced a deep learning model's prediction.
Scaling Monosemanticity
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet is the May 21
Shapley value
The Shapley value is a rule from cooperative game theory for dividing the payoff of a group among its members.
SmoothGrad
SmoothGrad is a saliency map technique that reduces visual noise in gradient-based explanations of neural network predictions by averaging gradients over many noisy copies of the input.
Sparse Coding
Sparse coding is a representation learning principle in which a signal is encoded as a linear combination of a small number of elements drawn from a (usually overcomplete) dictionary of atoms.
Sparse autoencoder
A sparse autoencoder (SAE) is a neural network that adds a sparsity penalty to an autoencoder's training loss so that only a small number of hidden units activate for any given input, producing a wide
Superposition (Mechanistic Interpretability)
Superposition is the phenomenon in which an artificial neural network represents more distinct features than it has dimensions in its activation space, by assigning those features to nearly-orthogonal (rather…
Towards Monosemanticity
Towards Monosemanticity is an October 2023 mechanistic interpretability paper from Anthropic that used a sparse autoencoder to decompose the internal activations of a small language model into thousands of…
Toy Models of Superposition
Toy Models of Superposition is a September 2022 mechanistic interpretability paper from Anthropic that shows how a neural network can represent more features than it has dimensions by packing them into…
Transcoder
A transcoder is a sparse neural network used in mechanistic interpretability research to approximate the input-to-output function of a component inside a transformer (most commonly an MLP sublayer) using a…
TransformerLens
TransformerLens is an open-source Python library for the mechanistic interpretability of GPT-style language models.
Variable importances
Variable importances, also called feature importances, are scores assigned to each input variable of a predictive model that measure how much that variable contributes to the model's output.
nnsight
nnsight is an open-source Python library for the interpretation and intervention of deep learning models, developed by the Bau Lab at Northeastern University.