Interpretability

Interpretability in artificial intelligence concerns what people can learn about a system's behavior, predictions, or internal computations, and whether that understanding is reliable enough for a stated purpose. The term has no single accepted definition. One influential definition describes interpretability as extracting relevant knowledge from a machine-learning model, while another survey separates transparency from post-hoc explanation and warns that the word can conceal several different goals.^[1][2] In this article, interpretability is the broad subject and explanation is a particular artifact, such as a rule, feature attribution, plot, example, or causal hypothesis.

Interpretability is not a property that can be assessed independently of audience and task. A concise reason code may be useful to a credit applicant, while a model developer may need a reproducible diagnostic and an auditor may need global evidence about failure modes. Frameworks therefore evaluate explanations against an intended user, question, and decision context instead of assuming that every readable explanation is adequate.^[1][3][4] Interpretability also does not, by itself, establish that a model is accurate, fair, safe, causal, or legally compliant.

Scope and terminology

Researchers use interpretability, explainability, transparency, and understandability inconsistently. Some use interpretability for models whose operation can be inspected directly and explainability for descriptions added after training. Others use the two terms interchangeably. Lipton's taxonomy, for example, separates kinds of transparency from post-hoc explanations, while Doshi-Velez and Kim organize evaluation by whether it uses a real application, simplified human tasks, or formal proxy tasks.^[2][3] These are useful frameworks, not universally binding definitions.

A practical analysis begins by stating the question. Common questions include:

Scroll sideways for more →

Three distinctions recur across the literature:

• Local and global: A local explanation concerns one input or a small neighborhood. A global explanation characterizes broader model behavior. Aggregating local explanations can provide a global summary, but it does not guarantee complete coverage.
• Intrinsic and post-hoc: An intrinsically interpretable model exposes its represented computation in a form intended for inspection. A post-hoc method analyzes a trained model and may approximate only selected behavior.
• Model-specific and model-agnostic: A model-specific method uses details such as gradients, tree structure, or activations. A model-agnostic method needs only queries to the model, although its perturbation scheme and data assumptions still matter.

NIST's four principles for explainable AI systems illustrate why producing an explanation is only a first step. The system should provide an explanation, make it meaningful to the intended recipient, ensure that it accurately reflects the relevant process, and operate within declared knowledge limits.^[4] NIST's AI Risk Management Framework separately notes that explainability and interpretability terminology varies and treats transparency, accountability, and validation as related but distinct concerns.^[5]

Interpretable models

An interpretable model exposes a representation that a person can inspect without first approximating it with a second model. Whether that representation is usable still depends on scale, feature meaning, transformations, interactions, and the intended reader.

Linear and logistic models

A linear regression prediction is a sum of an intercept and feature values multiplied by coefficients. This decomposition is exact for the fitted model. It does not make every coefficient a direct causal effect or a universally comparable importance score. Interpretation depends on feature units, coding, transformations, regularization, correlations among predictors, and which other variables are held fixed. For logistic regression, coefficients are additive on the log-odds scale before the link function is applied.

Standardizing inputs can make coefficient magnitudes more comparable within one fitted model, but it does not remove confounding or collinearity. A coefficient describes the model conditional on its specification and data; it does not by itself show how changing the corresponding real-world factor would change an outcome.

Decision trees and rules

A small decision tree can be read as nested conditions from root to leaf, and a short rule list can expose an ordered decision procedure. The computation represented by a path is exact. Interpretability decreases as trees become deep, rules proliferate, thresholds depend on opaque engineered features, or an ensemble combines many trees. Calling a tree-based system interpretable therefore requires reporting its size and the representation actually shown to users, not only its model family.

Generalized additive models

A generalized additive model represents a response through a link function:

g(E[y | x]) = beta_0 + sum_j f_j(x_j)

Each univariate shape function can be plotted, and the represented contributions sum exactly on the link scale. Pairwise interaction terms can be added explicitly, at the cost of a more complex display. Explainable Boosting Machines learn such shape functions with boosted shallow trees. One published evaluation found that these models were competitive with several black-box baselines on the tabular datasets it tested and retained exact decomposition of the fitted additive terms.^[22] That result is evidence for those tasks, not a universal claim that additive models match every complex model.

There is no theorem that predictive accuracy must always trade off against interpretability. Restricting a hypothesis class can hurt performance when a task needs interactions or structures the model cannot represent. On other datasets, a simpler or structured model may perform comparably, generalize better, or be easier to validate. Rudin has argued that high-stakes applications should consider developing interpretable models rather than defaulting to post-hoc explanations of black boxes.^[23] This is a methodological position, not proof that an adequate interpretable model exists for every task.

Post-hoc methods

Post-hoc methods answer selected questions about a trained machine learning system. Their output is an analysis of the model, not necessarily the process by which the model internally represented or computed a decision.

Permutation importance

Permutation importance measures how a chosen performance metric changes after values of one feature are permuted. It is model-agnostic and can be evaluated on held-out data. The result depends on the dataset, metric, permutation design, and fitted model. Correlated or substitutable predictors complicate interpretation: permuting one feature may create unrealistic combinations, and a model may rely on one of several redundant variables. Fisher, Rudin, and Dominici formalized related ideas as model reliance and emphasized that importance can vary across well-performing models, not only across variables in one model.^[9]

Impurity-based importance in tree ensembles is a different statistic derived from split improvements. It can be biased by feature cardinality and the opportunities a feature has to split. Coefficient magnitude is different again. These quantities should not be labeled simply "feature importance" without defining the estimand and scale.

Partial dependence, ICE, and ALE

A partial dependence plot averages a model's predictions while setting one or two selected features to a grid of values. The plot describes model behavior under that averaging procedure, not a causal dose-response curve. With correlated features, the procedure may evaluate combinations far outside the data distribution. It is more precise to say that partial dependence can extrapolate into unlikely regions than to say that the method literally assumes statistical independence.

Individual conditional expectation plots show one response curve per observation and can reveal heterogeneity hidden by an average. Crossing or diverging curves can indicate interactions in the fitted model, although the same off-distribution concern remains.^[10] Accumulated local effects use local conditional changes and integrate them across the feature distribution; Apley and Zhu developed ALE in part to reduce the extrapolation problem caused by correlated predictors.^[11] None of these plots proves that intervening on a feature would cause the displayed outcome change.

LIME

LIME explains a prediction by generating perturbed samples around an input, obtaining the black-box model's outputs, weighting samples by proximity, and fitting a sparse interpretable surrogate. The surrogate is intended to be faithful in the chosen neighborhood, not globally.^[6] The result depends on the interpretable representation, perturbation distribution, distance kernel, neighborhood width, surrogate family, and random seed.

A good LIME report therefore includes local-fidelity diagnostics and enough configuration to reproduce the result. A visually simple surrogate can be misleading if the sampled neighborhood is unrealistic or the black-box boundary cannot be approximated by the chosen model in that region. Repeating the analysis across seeds and plausible neighborhood definitions tests stability, but stable results still need a task-relevant fidelity test.

SHAP and Shapley-value methods

SHAP places several additive feature-attribution methods in a common framework. Within the framework defined by Lundberg and Lee, local accuracy, missingness, and consistency identify a unique additive attribution solution.^[7] For a given value function, Shapley values average a feature's marginal contribution over coalitions of other features.

The phrase "given value function" is important. To evaluate a feature as absent, an implementation must define a background or conditional distribution, masking operation, causal model, or other missingness semantics. Different choices answer different questions and can produce different attributions. Research on the many operational forms of Shapley values shows that the same game-theoretic name does not eliminate these modeling choices.^[8]

Local SHAP values decompose a particular model output relative to a baseline defined by the explainer. Summaries such as mean absolute SHAP values aggregate those local quantities over a sample. They do not automatically measure causal influence, fairness, or real-world actionability. Exact computation is exponential in the number of players in the general case, so practical systems use model-specific algorithms or approximations whose assumptions and error should be documented.

Gradients and saliency maps

Gradient attribution differentiates an output with respect to an input or internal activation. Integrated Gradients accumulates gradients along a straight-line path from a selected baseline to the input and was designed to satisfy sensitivity and implementation invariance properties.^[12] The attribution is relative to that baseline, so baseline choice is part of the explanation.

Grad-CAM weights feature maps in a convolutional layer with gradients for a target output to produce a coarse localization map.^[13] Such a heat map can show where a method assigns relevance, but visual plausibility is not a faithfulness test. Adebayo and colleagues proposed model-parameter and data-randomization sanity checks and found that some visually compelling saliency methods were insensitive to the trained parameters or labels in their experiments.^[14] Saliency studies should therefore test sensitivity to the model and data, compare meaningful controls, state the target and layer, and avoid treating a heat map as automatic evidence of a clinical or causal rationale.

Attention weights

Attention weights are internal quantities in transformer models, but they are not guaranteed explanations of a prediction. Jain and Wallace reported weak correspondence between standard attention weights and other importance measures in their tested natural-language models, and showed that substantially different attention distributions could yield similar outputs.^[15] Wiegreffe and Pinter argued that the conclusion depends on the definition of explanation and the tests applied, and proposed a more qualified evaluation framework.^[16]

The defensible conclusion is narrow: attention weights alone do not establish causal importance. An attention visualization can support a specific analysis when the relationship between the weights, values, downstream computation, and target behavior has been tested. It should not be described generically as what the model "looked at" or "reasoned about."

Counterfactual explanations and recourse

A counterfactual explanation identifies an alternative input that would change a model's output, often while minimizing a chosen distance from the original input. Wachter, Mittelstadt, and Russell proposed this form as a way to give outcome-focused information without exposing a model's full internals.^[17] The result is contrastive: under the fitted model, the displayed input differs and the output changes.

A counterfactual is not automatically an available action. It can prescribe an immutable feature, violate domain constraints, ignore dependencies between variables, or move to an input that is statistically possible but causally unreachable. Work on algorithmic recourse distinguishes changes to feature values from interventions a person can actually undertake and studies robustness to model uncertainty.^[18][19] Claims of actionable recourse require a domain model, feasibility constraints, uncertainty analysis, and attention to who bears the cost of the proposed change.

Probes

A probe is a supervised model trained on internal representations to predict an external label. If a linear probe predicts part-of-speech from a layer of BERT, the label is linearly decodable from those representations under the experiment. This does not by itself show that the original model uses that information for its task.

Probe capacity and the amount of supervision matter. A high-capacity probe may learn patterns from the probe dataset rather than expose a simple property of the representation. Control tasks, selectivity measures, minimum-description-length analyses, and interventions can help separate memorization, decodability, and causal use. Reviews of probing emphasize that these are distinct claims and that no single probe score resolves them.^[20][21]

Evaluating explanations

An explanation should be evaluated against the job it is supposed to do. Doshi-Velez and Kim distinguish application-grounded evaluation with domain experts and real tasks, human-grounded evaluation with simplified tasks and lay users, and functionally grounded evaluation using formal proxies without people.^[3] These levels answer different questions. A compactness metric cannot substitute for evidence that clinicians detect errors, and a favorable user rating cannot establish fidelity to a model.

Useful evaluation dimensions include:

Scroll sideways for more →

Murdoch and colleagues organize interpretable model analysis around predictive accuracy, descriptive accuracy, and relevancy. Descriptive accuracy concerns how faithfully an interpretation captures relationships learned by the model, while relevancy concerns whether it provides insight for a chosen audience and problem.^[1] NIST's explanation-accuracy and meaningfulness principles make a similar separation between reflecting the system and being understandable to a recipient.^[4]

A 2023 systematic review examined more than 300 papers introducing explainable-AI methods. It classified explanation quality through twelve properties, including correctness, completeness, consistency, continuity, contrastivity, compactness, context, and controllability. In the reviewed corpus, 33 percent relied exclusively on anecdotal evidence, 58 percent used quantitative evaluation, and 22 percent included a user study.^[25] Those figures describe the review's 2014-2020 sample and search criteria, not all interpretability research.

Human studies can contradict intuition. In preregistered experiments involving 3,800 participants, Poursabzi-Sangdeh and colleagues compared functionally identical regression models that varied in displayed transparency and feature count. A clear two-feature model helped participants simulate predictions, but did not make them follow beneficial predictions more closely; in the study's unusual cases, greater displayed transparency reduced their ability to detect and correct large model errors.^[24] This does not show that transparency is generally harmful. It shows that readability, error detection, reliance, and task performance are different outcomes.

Explanations can also generate over-reliance by presenting a plausible narrative for a wrong output. Trust is therefore not an appropriate universal optimization target. A better goal is calibrated reliance: people should be able to use a system when warranted, detect its limits, and override it when necessary. Studies should measure behavior and outcomes, not only whether participants report that an explanation looks convincing.

Law and regulation

Interpretability methods and legal explanation duties are related but not equivalent. The governing law specifies who must receive what information, under which decision process, and for what purpose. No statute cited here mandates LIME, SHAP, a saliency map, or another named technical method.

European Union

The General Data Protection Regulation requires specified information about automated decision-making in defined contexts. Articles 13, 14, and 15 refer to "meaningful information about the logic involved" and the significance and envisaged consequences when the relevant processing includes automated decision-making described in Article 22. Article 22 concerns decisions based solely on automated processing that produce legal or similarly significant effects, subject to exceptions and safeguards.^[26] It is therefore inaccurate to summarize the GDPR as an unrestricted general right to a technical explanation of every algorithmic decision.

In its February 27, 2025 judgment in Dun & Bradstreet Austria (C-203/22), the Court of Justice of the European Union held that the information under Article 15(1)(h) must explain the procedure and principles actually applied in a concise, transparent, intelligible, and accessible form so the data subject can understand how their data were used. The court also said that merely communicating a complex mathematical formula or every step of an algorithm would not necessarily satisfy that purpose.^[28]

The EU AI Act entered into force on August 1, 2024, but most of the provisions discussed here apply from August 2, 2026; Article 6(1) and its corresponding obligations apply from August 2, 2027. Article 13 requires applicable high-risk systems to be sufficiently transparent for deployers to interpret output and use it appropriately, and requires instructions containing specified information. From August 2, 2026, Article 86 gives an affected person a narrower right to obtain clear and meaningful explanations of the role of a high-risk system and the main elements of a decision when the decision is based on output from an Annex III high-risk system other than those in point 2 and produces legal effects or similarly significant adverse effects. Paragraph 2 recognizes exceptions or restrictions arising from Union or compliant national law, and paragraph 3 applies the article only to the extent that Union law does not otherwise provide the right.^[27] These provisions have defined scopes and should not be restated as a universal right to inspect a model.

United States credit decisions

The Equal Credit Opportunity Act requires a creditor taking adverse action to provide a statement of reasons or disclose the applicant's right to request one. When reasons are given, they must be specific.^[29] Regulation B requires the statement to identify the specific principal reasons for the action and states that indicating only that an applicant failed an internal standard or scoring system is insufficient.^[30]

These duties concern the actual reasons for the creditor's decision. They do not prescribe a particular machine-learning explanation algorithm. A post-hoc attribution may be inadequate if it does not faithfully represent the decision process or cannot produce the specific principal reasons required by the regulation. Legal review must consider the current rule, jurisdiction, facts, and workflow rather than assuming that a technically generated reason code establishes compliance.

Mechanistic interpretability

Mechanistic interpretability seeks testable accounts of internal computation in neural networks. It studies units, directions, learned features, attention heads, circuits, and other components, often with causal interventions. A "feature" in this literature is generally an operational object defined by a method, such as a direction or sparse latent that responds to a pattern. Human-readable examples can motivate a hypothesis, but the name assigned to a feature is not a formal proof of what it computes.

Superposition and sparse autoencoders

Toy-model work proposed superposition as a way a network can represent more features than it has dimensions when features are sparse, with interference as a cost.^[31] This is a mathematical and empirical model of representation, not a claim that every neuron has one fixed semantic meaning.

Sparse autoencoders are a form of learned dictionary learning applied to activations. They reconstruct an activation vector using a larger set of latent directions while encouraging sparse activation. Researchers inspect examples that activate a latent and test whether interventions on the latent affect behavior. Reconstruction quality, sparsity, dead latents, feature splitting, feature absorption, and evaluation selection all limit what can be inferred.

In Towards Monosemanticity, Anthropic studied activations in a one-layer transformer with a 512-neuron multilayer perceptron. It trained sparse autoencoders ranging from 512 to 131,072 features and analyzed a 4,096-feature run in detail. A blinded annotation experiment scored 412 activation intervals across 162 learned features and neurons.^[32]

In Scaling Monosemanticity, Anthropic trained sparse autoencoders with 1,048,576, 4,194,304, and 33,554,432 features on a middle-layer residual stream of Claude 3 Sonnet. Fewer than 300 features were active per token in the reported runs, and the autoencoders reconstructed at least 65 percent of activation variance. The percentage of dead features rose substantially with dictionary size. The authors selected interpretable examples and explicitly cautioned that those examples were not representative of all features.^[33] Steering the Golden Gate Bridge feature changed model behavior, providing causal evidence for that intervention, not a complete semantic account of the model.

OpenAI reported a 16-million-feature sparse autoencoder trained on GPT-4 activations in June 2024. Its report showed examples and quantitative scaling analyses, while stating that understanding the resulting features remained limited.^[34] Large dictionary size is therefore not a count of fully understood concepts.

Circuits and attribution graphs

A circuit is a hypothesized collection of components and interactions that implements behavior. Causal evidence can come from ablation, activation patching, path patching, feature steering, or other interventions. The evidence is scoped to the model, prompts, intervention, and measured output.

Anthropic's March 27, 2025 circuit-tracing work replaces selected model computations with a cross-layer-transcoder-based replacement model and generates attribution graphs over sparse features.^[35][37] The method paper reported that its small replacement model matched the original model's output roughly half the time under the authors' metric and, at release, did not model all query-key attention effects. Attribution graphs are therefore analyses of an imperfect proxy, not guaranteed transcripts of the original model's hidden reasoning.

The accompanying case studies used Claude 3.5 Haiku and reported experiments on language, planning, hallucination, arithmetic, and other behaviors. The authors stated that the graphs were incomplete, that the replacement model was an imperfect proxy, and that they obtained satisfying insight for about a quarter of prompts they tried; published examples were selected successes.^[36] Interventions can strengthen a specific hypothesis, but the study does not establish that all language-model computations are readable or that the method provides complete explanations.

On May 29, 2025, Anthropic released an open-source circuit-tracing library and a Neuronpedia-hosted interface for exploring attribution graphs on supported open-weight models.^[38] Tool availability does not remove the replacement-model and validation caveats described above.

Tools

Open-source libraries can calculate or visualize interpretability artifacts, but the user remains responsible for selecting a valid question, background distribution, target, layer, baseline, metric, and evaluation.

Scroll sideways for more →

A responsible report should record the library version, model and checkpoint, preprocessing, explained output, comparison baseline, background data, sampling and random seeds, approximation parameters, and evaluation result. Reproducibility metadata is part of the explanation's evidence.

Practical checklist

Before relying on an interpretation:

1. Name the question. Separate description, prediction, causal effect, debugging, recourse, compliance, and human decision support.
2. Name the audience and action. Specify what a developer, affected person, auditor, or operator should be able to do.
3. Choose an estimand. Define what "importance," "absence," "local," "similar," or "counterfactual" means.
4. Inspect data support. Flag extrapolation, correlated features, unrealistic perturbations, and out-of-distribution inputs.
5. Measure fidelity. Use exact decompositions where available and task-specific perturbations or interventions otherwise.
6. Test stability and alternatives. Vary seeds, baselines, backgrounds, neighborhoods, and plausible methods.
7. Separate plausibility from causality. A readable heat map, feature name, or narrative is a hypothesis unless causal evidence supports the claim.
8. Evaluate people on the real task. Measure error detection, calibration, decision quality, and burden, not only preference.
9. Expose uncertainty and limits. Report approximation error, coverage, unsupported regions, and known failure modes.
10. Keep governance separate. Explanation is one input to validation, fairness analysis, safety review, documentation, and legal compliance.

Explain like I'm 5

Imagine a calculator that gives an answer and also shows some of its work. Seeing the work can help you check it, but only if the steps are really connected to the answer. A neat-looking story might still be wrong.

Some computer models are built from short rules that people can read. Others are too large to inspect directly, so researchers use tests that highlight inputs, compare nearby cases, or change parts of the model. Each test answers a particular question. Interpretability means deciding which question matters, checking that the test tells the truth about the model, and showing the result in a form the right person can use.

See also

• Explainable AI
• Feature engineering
• Algorithmic fairness and bias
• AI safety
• Deep neural network
• Random forest
• Gradient boosting

References

1. ^Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., and Yu, B. "Definitions, methods, and applications in interpretable machine learning." Proceedings of the National Academy of Sciences 116(44), 2019. Article
2. ^Lipton, Z. C. "The Mythos of Model Interpretability." Communications of the ACM 61(10), 2018. arXiv manuscript
3. ^Doshi-Velez, F., and Kim, B. "Towards A Rigorous Science of Interpretable Machine Learning." 2017. arXiv manuscript
4. ^Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., and Przybocki, M. A. "Four Principles of Explainable Artificial Intelligence." NISTIR 8312, 2021. NIST publication
5. ^National Institute of Standards and Technology. "AI Risks and Trustworthiness." AI Risk Management Framework resources. NIST AI RMF resource
6. ^Ribeiro, M. T., Singh, S., and Guestrin, C. "'Why Should I Trust You?': Explaining the Predictions of Any Classifier." Proceedings of KDD, 2016. arXiv manuscript
7. ^Lundberg, S. M., and Lee, S.-I. "A Unified Approach to Interpreting Model Predictions." Advances in Neural Information Processing Systems 30, 2017. Proceedings paper
8. ^Sundararajan, M., and Najmi, A. "The Many Shapley Values for Model Explanation." Proceedings of ICML, 2020. PMLR paper
9. ^Fisher, A., Rudin, C., and Dominici, F. "All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously." Journal of Machine Learning Research 20(177), 2019. JMLR paper
10. ^Goldstein, A., Kapelner, A., Bleich, J., and Pitkin, E. "Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation." Journal of Computational and Graphical Statistics 24(1), 2015. arXiv manuscript
11. ^Apley, D. W., and Zhu, J. "Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models." Journal of the Royal Statistical Society: Series B 82(4), 2020. Journal article
12. ^Sundararajan, M., Taly, A., and Yan, Q. "Axiomatic Attribution for Deep Networks." Proceedings of ICML, 2017. PMLR paper
13. ^Selvaraju, R. R., et al. "Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization." Proceedings of ICCV, 2017. CVF paper
14. ^Adebayo, J., et al. "Sanity Checks for Saliency Maps." Advances in Neural Information Processing Systems 31, 2018. Proceedings paper
15. ^Jain, S., and Wallace, B. C. "Attention is not Explanation." Proceedings of NAACL-HLT, 2019. ACL Anthology paper
16. ^Wiegreffe, S., and Pinter, Y. "Attention is not not Explanation." Proceedings of EMNLP-IJCNLP, 2019. ACL Anthology paper
17. ^Wachter, S., Mittelstadt, B., and Russell, C. "Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR." Harvard Journal of Law and Technology 31(2), 2018. arXiv manuscript
18. ^Ustun, B., Spangher, A., and Liu, Y. "Actionable Recourse in Linear Classification." Proceedings of the Conference on Fairness, Accountability, and Transparency, 2019. arXiv manuscript
19. ^Karimi, A.-H., Barthe, G., Schölkopf, B., and Valera, I. "A Survey of Algorithmic Recourse: Contrastive Explanations and Consequential Recommendations." ACM Computing Surveys 55(5), 2023. arXiv manuscript
20. ^Belinkov, Y. "Probing Classifiers: Promises, Shortcomings, and Advances." Computational Linguistics 48(1), 2022. ACL Anthology paper
21. ^Hewitt, J., and Liang, P. "Designing and Interpreting Probes with Control Tasks." Proceedings of EMNLP-IJCNLP, 2019. ACL Anthology paper
22. ^Nori, H., Caruana, R., Bu, Z., Shen, J. H., and Kulkarni, J. "Accuracy, Interpretability, and Differential Privacy via Explainable Boosting." Proceedings of ICML, 2021. PMLR paper
23. ^Rudin, C. "Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead." Nature Machine Intelligence 1, 2019. Journal article
24. ^Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., and Wallach, H. "Manipulating and Measuring Model Interpretability." Proceedings of CHI, 2021. Author manuscript
25. ^Nauta, M., et al. "From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI." ACM Computing Surveys 55(13s), 2023. arXiv manuscript
26. ^European Parliament and Council. Regulation (EU) 2016/679, General Data Protection Regulation, Articles 13, 14, 15, and 22. Official consolidated text
27. ^European Parliament and Council. Regulation (EU) 2024/1689, Artificial Intelligence Act, Articles 13, 86, and 113. Official text
28. ^Court of Justice of the European Union. "Dun & Bradstreet Austria," Case C-203/22, judgment of February 27, 2025. Official judgment
29. ^United States Code. 15 U.S.C. Section 1691(d), Equal Credit Opportunity Act adverse-action notice requirements. Official U.S. Code, 2023 edition
30. ^Consumer Financial Protection Bureau. Regulation B, 12 CFR Part 1002, Section 1002.9. Official regulation
31. ^Elhage, N., et al. "Toy Models of Superposition." Transformer Circuits, 2022. Research report
32. ^Bricken, T., et al. "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning." Transformer Circuits, 2023. Research report
33. ^Templeton, A., et al. "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet." Transformer Circuits, 2024. Research report
34. ^Gao, L., et al. "Scaling and evaluating sparse autoencoders." OpenAI, 2024. OpenAI report
35. ^Lindsey, J., et al. "Circuit Tracing: Revealing Computational Graphs in Language Models." Transformer Circuits, 2025. Methods report
36. ^Lindsey, J., et al. "On the Biology of a Large Language Model." Transformer Circuits, 2025. Case-study report
37. ^Anthropic. "Tracing the thoughts of a large language model." March 27, 2025. Research announcement
38. ^Anthropic. "Open-sourcing circuit tracing tools." May 29, 2025. Research announcement
39. ^SHAP contributors. "SHAP." Official GitHub repository
40. ^Ribeiro, M. T., et al. "lime." Official GitHub repository
41. ^PyTorch contributors. "Captum: Model Interpretability for PyTorch." Official documentation
42. ^InterpretML contributors. "InterpretML." Official GitHub repository
43. ^TransformerLens contributors. "TransformerLens." Official GitHub repository
44. ^circuit-tracer contributors. "circuit-tracer." Maintainer repository
45. ^SeldonIO. "Alibi Explain." Official GitHub repository