Integrated Gradients
Integrated Gradients (IG) is a feature-attribution method for explainable AI that explains a neural network prediction by assigning each input feature an importance score, computed by integrating the model's gradients along a straight-line path from a neutral reference input (the baseline) to the actual input. It was introduced by Mukund Sundararajan, Ankur Taly, and Qiqi Yan in the 2017 ICML paper "Axiomatic Attribution for Deep Networks," which derives the method from two axioms, Sensitivity and Implementation Invariance, that the authors argue every attribution method ought to satisfy. [1] Unlike earlier saliency map techniques that read off raw gradients at a single point, IG averages gradients over the whole path, and that integration step is what gives the method its name.
The paper frames IG as both principled and lightweight. In its own words, the authors "identify two fundamental axioms, Sensitivity and Implementation Invariance, that attribution methods ought to satisfy," then "use the axioms to guide the design of a new attribution method called Integrated Gradients" that "requires no modification to the original network and is extremely simple to implement; it just needs a few calls to the standard gradient operator." [1] IG has become one of the most widely cited gradient-based attribution techniques for deep learning models, and although it is not the only method that satisfies both axioms, the paper shows it is the unique path method that is also symmetry-preserving. [1]
IG sits within the broader field of attribution methods and is closely tied to the Aumann-Shapley value from cooperative game theory, which generalizes the discrete Shapley value to continuous settings. It has been adopted by major libraries including Captum for PyTorch, the Saliency library for TensorFlow, Alibi, and SHAP. It has been applied to image classification on networks like Inception and ResNet trained on ImageNet, to natural language tasks with BERT and other transformer models, to tabular health-care and gene-expression data, and to chemistry and drug-discovery models. [1][10][15]
What is Integrated Gradients?
Integrated Gradients is a model interpretability technique that quantifies how much each input feature contributed to a particular prediction, measured relative to a chosen baseline. The core idea is to interpolate between the baseline and the input, compute the model's gradient at many points along that straight line, average those gradients, and scale the result by the input-minus-baseline difference. The output is one attribution score per feature, with positive scores indicating features that pushed the prediction up and negative scores indicating features that pushed it down.
The method was designed to fix a known failure of plain gradients. Raw saliency can report zero importance for a feature that clearly matters, because the model output may be locally flat (saturated) at the exact input point even though the feature changed the prediction overall. By integrating gradients across the path rather than sampling a single point, IG guarantees that any feature responsible for the prediction difference receives credit. The trade-off is cost: where vanilla saliency needs one backward pass, IG needs many (the original paper recommends 20 to 300). [1][3]
How does Integrated Gradients work? (mathematical formulation)
Let F: R^n -> R be a differentiable function representing a neural network output (for example, a class probability or logit). Let x be the input of interest and x' be a chosen baseline. The Integrated Gradient for input feature i is defined as the path integral of the partial derivative of F with respect to feature i, taken along the straight-line interpolation between x' and x:
IG_i(x) = (x_i - x'_i) * integral from alpha=0 to 1 of dF(x' + alpha * (x - x')) / dx_i d(alpha)
In words: scale each input dimension by the difference between the input and baseline values, and multiply by the average partial derivative of the model along the linear path between them. The factor (x_i - x'_i) ensures the attribution is zero when an input feature equals the baseline, and the integral averages out local fluctuations in the gradient caused by saturated activations or sharp nonlinearities. [1]
How is the integral approximated numerically?
The integral is rarely computed analytically. In practice, IG is approximated by a Riemann sum with m discrete steps:
IG_i(x) ~ (x_i - x'_i) / m * sum from k=1 to m of dF(x' + (k/m) * (x - x')) / dx_i
Each step requires one forward pass and one backward pass through the network. The original paper recommends 20 to 300 steps, enough to approximate the integral to within about 5 percent, with the exact number depending on how much the model output changes between the baseline and input. [1][3] The TensorFlow IG tutorial, which uses Inception V1 on ImageNet in the spirit of the original paper, sets 50 steps and notes that in practice the count can be in the thousands to approximate the integral accurately; Captum also defaults to 50 steps. [2][3]
A simple worked example
Consider a one-dimensional ReLU network F(x) = max(0, x - 0.5) evaluated at x = 1.0 with baseline x' = 0.0. The model output is F(1.0) = 0.5 and F(0.0) = 0.0. The gradient dF/dx is 0 for x < 0.5 and 1 for x > 0.5. Along the path from 0 to 1 the gradient equals 1 for half the path and 0 for the other half, so the path-averaged gradient is 0.5. Multiplying by (x - x') = 1 gives an attribution of 0.5, exactly equal to F(x) - F(x'). Vanilla saliency, by contrast, would report a gradient of 1 evaluated at x = 1.0, ignoring the saturated region between 0 and 0.5 where the function was actually flat.
What are the axioms behind Integrated Gradients?
The central contribution of the IG paper is the introduction of two axioms that any attribution method should satisfy. The authors argue that prior literature had judged methods primarily by visual appeal rather than by formal properties, making it hard to know whether a saliency map captures model behavior or merely produces plausible-looking heatmaps. As the paper puts it, most known attribution methods do not satisfy these axioms, "which we consider to be a fundamental weakness of those methods." [1]
What is the Sensitivity axiom?
The Sensitivity (a) axiom states: if an input x and baseline x' differ in a single feature, and the model output differs between them, then that feature must receive a non-zero attribution. The Sensitivity (b) axiom adds the converse condition that if a function does not mathematically depend on a variable, then attribution to that variable should be zero.
Vanilla saliency map methods, which compute dF/dx at the input point alone, can fail Sensitivity (a) because of saturated activations. In deep networks with many ReLU units, the gradient of a class score with respect to an input pixel can be exactly zero even when changing the pixel from black to its observed value would change the prediction. IG circumvents this by averaging gradients along the interpolation path, which guarantees that any feature contributing to the prediction difference receives non-zero credit. [1]
What is the Implementation Invariance axiom?
The Implementation Invariance axiom requires that two networks computing the exact same input-output function F should produce identical attributions, even if they have different internal architectures. Two networks are functionally equivalent if F_1(x) = F_2(x) for every input x.
This axiom is satisfied automatically by methods that depend only on input gradients, since the gradient of F is a property of the function, not of its implementation. Vanilla gradients and IG both satisfy it. DeepLIFT, proposed by Shrikumar, Greenside, and Kundaje, replaces the partial derivative with a reference difference quotient and propagates modified gradients backward through the network. [4] Because DeepLIFT rules depend on the choice of nonlinearity and on how layers are decomposed, two functionally equivalent networks with different layer structures can yield different attributions. The same critique applies to LRP (Bach et al. 2015). [5]
IG is therefore distinguished by satisfying both axioms simultaneously. Sundararajan and colleagues prove that, among the family of path integration methods that integrate gradients along some path from baseline to input, the linear path is the unique choice that satisfies a stronger set of axioms including Symmetry-Preserving (symmetric inputs receive equal attribution). [1]
What is the Completeness property and how does it connect to Shapley values?
IG satisfies a useful accounting property called Completeness: the per-feature attributions sum exactly to the difference in model output between the input and the baseline.
sum over i of IG_i(x) = F(x) - F(x')
This is a direct consequence of the fundamental theorem of calculus applied to the path integral, and it means the attributions can be interpreted as a decomposition of the prediction change into per-feature contributions. Completeness also makes IG comparable to SHAP and other attribution methods that produce a budgeted distribution of credit. In practice, Completeness doubles as a built-in correctness check: summing the attributions and comparing against F(x) - F(x') reveals whether the Riemann approximation used enough steps. [6]
The connection to game theory runs deeper than the surface analogy. IG is mathematically equivalent to the Aumann-Shapley value for the linear path between baseline and input. Aumann and Shapley developed this in 1974 as a continuous extension of the discrete Shapley value for cost allocation where players contribute fractional amounts. [7] In game-theoretic terms, IG distributes the payout F(x) - F(x') among features in a way that satisfies Efficiency, Linearity, Symmetry, and the Dummy axioms.
Sundararajan and Najmi's 2020 paper "The many Shapley values for model explanation" later clarified the relationship between IG, KernelSHAP, baseline Shapley, and other Shapley-derived methods, showing they correspond to different choices of game formulation. [8] IG's specific choice of a deterministic linear path makes it computationally tractable in continuous settings where a true expectation over feature subsets would require exponential evaluation.
How do you choose a baseline?
The choice of baseline x' is the most consequential modeling decision in IG and the most discussed practical issue in the literature. The baseline defines what "absent" or "neutral" features mean. Different baselines produce different attributions, sometimes dramatically so.
The original paper recommends that the baseline should produce a near-zero output: F(x') ~ 0. For an image classifier this typically means that a baseline image should not be predicted as the target class. Common baseline choices include:
| Baseline type | Description | Typical use case |
|---|---|---|
| Zero / black image | Input replaced with all zeros (or zero after normalization) | Default for image models, often used with Inception, ResNet |
| Random noise | Pixels sampled from Gaussian or uniform noise | Avoids systematic bias of black baseline; common alternative |
| Gaussian blur | Heavily blurred version of the input | Removes high-frequency content but preserves overall structure |
| Mean image | Average of training set | Represents an "average" example |
| Multiple baselines | Average IG over many baselines | Reduces baseline sensitivity, used in Expected Gradients |
| Padding token | Embedding of a special PAD or [MASK] token | Standard for transformer NLP models |
| Empty string embedding | Zero vector in embedding space | Alternative for text classification |
The black-image baseline has been criticized because it is itself informative: a black region in a natural image is unusual, and the gradient with respect to a black pixel may emphasize edge detectors that fire on dark regions rather than truly indicating absence of the feature. [9] Many practitioners now use noise or multi-baseline approaches.
Erion and colleagues introduced Expected Gradients (first posted in 2019 and published in 2021), which marginalizes over a distribution of baselines (typically the training distribution) by averaging IG attributions across sampled baselines. [10] The SHAP library describes this as reformulating the IG integral as an expectation over reference values sampled from a background dataset, so that the attributions approximate SHAP values; it also removes the need to pick a single arbitrary baseline. [31]
For text models, the baseline is usually the embedding of a PAD or zero token, since the model itself only sees continuous embeddings. Attribution is computed with respect to the embedding vectors and then summed across embedding dimensions to give a scalar score per token.
How is Integrated Gradients implemented in practice?
A reference implementation of IG looks like this in pseudocode:
def integrated_gradients(model, x, x_prime, target_class, m=50):
alphas = linspace(0, 1, m)
interpolated = x_prime + alphas[:, None] * (x - x_prime)
grads = []
for x_alpha in interpolated:
x_alpha.requires_grad = True
output = model(x_alpha)[target_class]
grad = autograd.grad(output, x_alpha)[0]
grads.append(grad)
avg_grad = mean(grads, axis=0)
attributions = (x - x_prime) * avg_grad
return attributions
In production the interpolated inputs are batched, so a single forward pass computes outputs for all m points at once. Memory becomes a constraint at large m, and implementations often chunk the integration into mini-batches.
A common diagnostic is to verify completeness numerically: sum the attributions and compare against F(x) - F(x'). Captum exposes this via the return_convergence_delta=True flag in its IntegratedGradients class. Deltas above a few percent indicate the model contains discontinuities or that more steps are required.
IG requires the model to be differentiable along the path. For piecewise-linear networks (ReLU, max-pooling) the path is differentiable almost everywhere, and the Riemann sum converges. For non-differentiable operations like argmax, gradient smoothing tricks are needed.
What is Integrated Gradients used for? (applications)
IG has been deployed across many domains. The table below summarizes representative use cases.
| Domain | Model architecture | Use of IG | Reference |
|---|---|---|---|
| Image classification | GoogleNet (Inception) on ImageNet; diabetic retinopathy CNN | Pixel-level saliency for class predictions | Sundararajan et al. 2017 [1] |
| NLP / question answering | Visual, tabular, and reading-comprehension QA models | Word-level attribution showing models ignore important question terms; used to craft adversarial questions | Mudrakarta et al. 2018 [11] |
| Gene expression | Two-layer neural networks on AML patient gene expression | Expected Gradients with a graph attribution prior for anti-cancer drug response prediction | Erion et al. 2021 [10] |
| Healthcare | Neural networks on tabular patient data | Expected Gradients with a sparsity prior for 10-year mortality prediction | Erion et al. 2021 [10] |
| Genomics | BPNet (convolutional) | Base-resolution contribution scores from DeepLIFT, a related method, used with TF-MoDISco for motif discovery | Avsec et al. 2021 [13] |
| Medical imaging | Diabetic retinopathy CNN | Principled visualization of IG attributions for diagnosis | Sundararajan et al. 2019 [14] |
| Drug discovery | Graph neural networks | Atom-level attribution for molecular property prediction | Jimenez-Luna et al. 2020 [15] |
| Audio event identification | AudioSet audio classifier | Blur IG attributions for audio events | Xu et al. 2020 [9] |
In image work, IG is usually visualized as a heatmap overlay, with positive attributions in red and negative in blue. The maps tend to be smoother than vanilla saliency because integration averages out local gradient noise. Common practice is to take absolute attributions or sum across color channels.
For NLP tasks, IG attributions to token embeddings are summed across embedding dimensions to give a scalar score per token, visualized as heatmap-colored text. Researchers have used IG to find erroneous model logic: Mudrakarta and colleagues showed that question-answering models often ignore important question terms and used this to craft adversarial questions, and the original paper found a question-classification model using "charles" as a trigger for yes/no questions. [1][11]
In genomics, attribution methods closely related to IG are used to extract sequence motifs from convolutional networks trained on regulatory tasks; BPNet, for example, computes base-resolution contribution scores with DeepLIFT and summarizes them into transcription factor binding motifs with TF-MoDISco. [13]
What variants and extensions of Integrated Gradients exist?
Several extensions of IG have been proposed to address specific limitations.
Expected Gradients (Erion et al. 2021) replaces the single baseline with a distribution sampled from a reference set, often the training data. Averaging over these baselines lets the attributions approximate SHAP values. [31] The approach reduces baseline sensitivity and can be used as an attribution-guided regularizer during training. [10]
Blur Integrated Gradients (Xu et al. 2020) replaces the linear path with a path through progressively blurred versions of the input. The intuition is that a Gaussian-blurred image is a more semantically meaningful "absence of features" baseline than a black image. The paper argues that the blur path uses perturbations free of artifacts, produces cleaner explanations, reveals at what scale a network recognizes an object, and removes the need to choose a baseline. [9]
Guided Integrated Gradients (Kapishnikov et al. 2021) chooses the integration path adaptively rather than using a straight line. At each step, the path moves only the subset of features with the lowest absolute partial derivatives (for example the bottom 10%) closer to their input values, rather than moving all features uniformly. Empirically, Guided IG produces less noisy saliency maps on image classifiers, and as a path method it still satisfies completeness, though it gives up the straight-line path that defines the Aumann-Shapley form. [17]
SmoothGrad-Integrated Gradients combines IG with SmoothGrad by averaging IG over multiple noisy versions of the input. [18] This reduces visual noise in the saliency map at the cost of additional compute. The combined method is sometimes called Smooth IG or SmoothGrad-IG.
Integrated Hessians (Janizek, Sturmfels, Lee 2021) extends IG to capture pairwise feature interactions by integrating second derivatives along the path. The result is a matrix of pairwise attributions that decomposes the prediction not just into per-feature credit but into per-feature-pair interactions, useful for diagnosing how a model combines features. [19]
XRAI (Kapishnikov et al. 2019) combines IG with image segmentation. After computing pixel-level IG, XRAI aggregates attributions over superpixel regions to produce region-level explanations that are more interpretable on natural images than pixel-level heatmaps. [20]
What are the limitations and critiques of Integrated Gradients?
Despite its theoretical grounding, IG has documented limitations. The most influential critique is the 2018 NeurIPS paper "Sanity Checks for Saliency Maps" by Adebayo and colleagues. [21] They proposed two tests: a parameter randomization test, where weights are randomized layer by layer, and a data randomization test, where labels are shuffled. A reliable method should produce attributions that change substantially when the model is randomized. They found that several popular methods produced visually similar attributions on randomized and trained networks: Guided Backprop and Guided GradCAM were invariant to higher-layer weights, and IG and Gradient x Input maps stayed visually similar because the input structure dominated them, although randomization disrupted the sign of their attributions. The conclusion was that visual similarity to edge maps does not imply faithfulness to the model. This sparked more rigorous evaluation methodology across the saliency literature.
Other limitations include:
Path dependence. The straight-line path is one of many. The Aumann-Shapley value depends on the chosen path, and although the linear path is the unique symmetric one, alternatives (Blur IG, Guided IG) yield different attributions. The choice rests on axiomatic arguments and visual quality, not pure empirics.
Computational cost. A single IG attribution requires 20 to 300 forward and backward passes. For a large transformer with billions of parameters, this can be prohibitive. Approximate methods like sampling-based IG reduce cost.
Baseline sensitivity. As discussed above, attributions can change significantly with the choice of baseline. There is no universally correct baseline, and different fields have developed domain-specific conventions.
Discrete inputs. For categorical or token inputs the partial derivative is undefined. The workaround is to compute IG with respect to the continuous embedding and sum across embedding dimensions. The attribution is then over the embedding, not over the original token.
Limited feature interactions. Standard IG produces an additive decomposition into per-feature contributions. Real models combine features non-additively, and this structure is invisible in standard IG. Integrated Hessians and other extensions address this gap.
Adversarial fragility. Small, perceptually indistinguishable perturbations that leave the predicted label unchanged can produce dramatically different IG attributions, so an IG map can be manipulated without changing the model's output. [22]
Faithfulness vs intuitiveness. Visually appealing saliency maps are not always faithful to the model. How faithful IG is compared with other methods depends on the evaluation used. [21]
How does Integrated Gradients compare with other attribution methods?
The following table compares IG with other widely used attribution methods. All of them produce per-feature attributions but differ in how they obtain those scores.
| Method | Year | Approach | Implementation invariant | Completeness | Computational cost |
|---|---|---|---|---|---|
| Vanilla Saliency | 2014 | Gradient at input point | Yes | No | 1 backward pass |
| Guided Backprop | 2014 | Gradient with negative values clipped | No | No | 1 backward pass |
| Class Activation Map (CAM) | 2016 | Weighted sum of last-layer activations | No | No | 1 forward pass |
| Grad-CAM | 2017 | Gradient-weighted activation map | Partially | No | 1 backward pass |
| LRP | 2015 | Layer-wise relevance propagation rules | No | Yes (with constraints) | 1 backward pass |
| DeepLIFT | 2017 | Reference-based modified backprop | No | Yes | 1 backward pass |
| Integrated Gradients | 2017 | Path integral of gradients | Yes | Yes | 20 to 300 passes |
| SmoothGrad | 2017 | Average gradients over noisy inputs | Yes | No | n forward+backward passes |
| LIME | 2016 | Local linear model around input | Model agnostic | Approximately | Many forward passes |
| SHAP (KernelSHAP) | 2017 | Sampled Shapley values | Model agnostic | Yes | Many forward passes |
| GradientSHAP | 2017 | IG averaged over baselines and noise | Yes | Yes | Many passes |
| Expected Gradients | 2019 | IG averaged over baseline distribution | Yes | Yes | Many passes |
| Blur IG | 2020 | IG along Gaussian blur path | Yes | Yes | 20 to 300 passes |
| Guided IG | 2021 | IG along adaptively chosen path | Yes | Yes | 20 to 300 passes |
Saliency map methods (Simonyan et al. 2014) compute the absolute value of dF/dx at the input. They are cheap but suffer from gradient saturation and noise. [23]
DeepLIFT (Shrikumar et al. 2017) computes attributions by propagating reference activations backward through the network using rules that depend on the specific layer types. DeepLIFT is faster than IG (one backward pass vs many), but it sacrifices Implementation Invariance. [4]
LRP (Bach et al. 2015) propagates relevance backward through the network using conservation rules. Like DeepLIFT, it depends on the specific layer types and so is not implementation invariant. [5]
SHAP (Lundberg and Lee 2017) is a unifying framework that estimates the Shapley value of each input feature using sampling or kernel-based techniques. SHAP is model-agnostic, but its sampling cost grows quickly with the number of features. The SHAP library's GradientExplainer implements expected gradients, an extension of IG. [31]
LIME (Ribeiro et al. 2016) fits a sparse linear model in the neighborhood of the input and uses its coefficients as attributions. LIME is also model-agnostic and works for any classifier, but is more sensitive to neighborhood definition and sampling. [24]
SmoothGrad (Smilkov et al. 2017) reduces gradient noise by averaging gradients over many noisy versions of the input. It is often combined with other gradient methods including IG to denoise the attribution maps. [18]
Which libraries implement Integrated Gradients?
IG is supported by many interpretability libraries.
Captum is the official PyTorch interpretability library from Meta. Its IntegratedGradients class supports batched integration, multiple baselines, and convergence diagnostics. Captum also includes LayerIntegratedGradients for attributing to intermediate layer activations. [25]
TensorFlow Saliency library (Google's PAIR team) provides a NumPy-based implementation that interoperates with TensorFlow models. It also includes XRAI, Blur IG, and Guided IG. [26]
Alibi is a Python library focused on machine learning explanation, with IntegratedGradients built on top of TensorFlow and Keras. [27]
SHAP library includes a GradientExplainer class that implements expected gradients: it reformulates the IG integral as an expectation and samples reference values from a background dataset. [31]
InterpretML (Microsoft) is a related interpretability toolkit, but it focuses on glassbox models and black-box explainers (SHAP Kernel Explainer, LIME, Morris sensitivity analysis, partial dependence) and does not include IG. [28]
All implementations share the same algorithm: take an input and a baseline, generate m interpolated inputs along the linear path, compute gradients at each, and combine them with a numerical integration rule (a Riemann sum, or Gauss-Legendre quadrature, which is Captum's default) scaled by (x - x'). Differences arise in batching, layer-level attribution, and convergence diagnostics. For large transformers, layer-level IG is often more useful than input-level IG because token IDs have no natural baseline. Layer IG attributes to embedding outputs or attention activations, which admit natural zero baselines.
ELI5: Integrated Gradients in plain terms
Imagine a model that looks at a photo and decides "this is a cat." You want to know which pixels convinced it. The naive approach is to nudge each pixel a tiny bit and see how much the "cat" score wiggles, but that can be misleading: a pixel might already be "maxed out," so wiggling it does nothing locally even though it was important. Integrated Gradients fixes this by starting from a blank reference image (the baseline, often all black) and slowly fading the real photo in, checking at each step how the "cat" score responds. Adding up those responses tells you how much each pixel mattered on the journey from blank to real. A nice bonus is that all the per-pixel scores add up exactly to the change in the "cat" score between the blank image and the real one, so nothing is double-counted or lost.
Continue reading
- SHAP (SHapley Additive exPlanations)SHAP (SHapley Additive exPlanations) is a game-theoretic method that explains an individual machine learning prediction by assigning each input feature a numerical value representing how much it pushed that…Mentioned 11 times above
- Shapley valueThe Shapley value is a rule from cooperative game theory for dividing the payoff of a group among its members.Mentioned 9 times above
- DeepLIFTDeepLIFT (Deep Learning Important FeaTures) is a feature attribution method for deep neural networks introduced by Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje at Stanford University in 2017.Mentioned 8 times above
- Saliency mapA saliency map is an explainable AI visualization that highlights which parts of an input, most often the individual pixels of an image, most influenced a deep learning model's prediction.Mentioned 8 times above
- SmoothGradSmoothGrad is a saliency map technique that reduces visual noise in gradient-based explanations of neural network predictions by averaging gradients over many noisy copies of the input.Mentioned 5 times above
- InterpretabilityInterpretability in artificial intelligence concerns what people can learn about a system's behavior, predictions, or internal computations, and whether that understanding is reliable enough for a stated…Mentioned 4 times above
More from See also
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12Sundararajan, M., Taly, A., & Yan, Q. (2017). "Axiomatic Attribution for Deep Networks." *Proceedings of the 34th International Conference on Machine Learning (ICML)*. arxiv.org/...1703.01365
- ^Captum documentation. "Integrated Gradients." Meta AI. captum.ai/...integrated_gradients
- ^1 ^2 ^3TensorFlow tutorials. "Integrated Gradients." tensorflow.org/...integrated_gradients
- ^1 ^2Shrikumar, A., Greenside, P., & Kundaje, A. (2017). "Learning Important Features Through Propagating Activation Differences." *ICML 2017*. arxiv.org/...1704.02685
- ^1 ^2Bach, S., Binder, A., Montavon, G., Klauschen, F., Muller, K. R., & Samek, W. (2015). "On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation." *PLOS ONE*. journals.plos.org/...article
- ^Lundberg, S. M., & Lee, S. I. (2017). "A Unified Approach to Interpreting Model Predictions." *NeurIPS 2017*. arxiv.org/...1705.07874
- ^Aumann, R. J., & Shapley, L. S. (1974). *Values of Non-Atomic Games*. Princeton University Press. press.princeton.edu/...values-of-non-atomic-games
- ^Sundararajan, M., & Najmi, A. (2020). "The Many Shapley Values for Model Explanation." *ICML 2020*. arxiv.org/...1908.08474
- ^1 ^2 ^3Xu, S., Venugopalan, S., & Sundararajan, M. (2020). "Attribution in Scale and Space." *CVPR 2020*. arxiv.org/...2004.03383
- ^1 ^2 ^3 ^4 ^5Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M., & Lee, S. I. (2021). "Improving Performance of Deep Learning Models with Axiomatic Attribution Priors and Expected Gradients." *Nature Machine Intelligence*. arxiv.org/...1906.10670
- ^1 ^2Mudrakarta, P. K., Taly, A., Sundararajan, M., & Dhamdhere, K. (2018). "Did the Model Understand the Question?" *ACL 2018*. arxiv.org/...1805.05492
- Lundberg, S. M., Nair, B., Vavilala, M. S., et al. (2018). "Explainable Machine-Learning Predictions for the Prevention of Hypoxaemia During Surgery." *Nature Biomedical Engineering*. nature.com/...s41551-018-0304-0
- ^1 ^2Avsec, Z., Weilert, M., Shrikumar, A., et al. (2021). "Base-Resolution Models of Transcription-Factor Binding Reveal Soft Motif Syntax." *Nature Genetics*. nature.com/...s41588-021-00782-6
- ^Sundararajan, M., Xu, S., Taly, A., Sayres, R., & Najmi, A. (2019). "Exploring Principled Visualizations for Deep Network Attributions." *IUI Workshops (ExSS 2019)*. ceur-ws.org/...IUI19WS-ExSS2019-16.pdf
- ^1 ^2Jimenez-Luna, J., Grisoni, F., & Schneider, G. (2020). "Drug Discovery with Explainable Artificial Intelligence." *Nature Machine Intelligence*. nature.com/...s42256-020-00236-4
- Becker, S., Ackermann, M., Lapuschkin, S., Muller, K. R., & Samek, W. (2018). "Interpreting and Explaining Deep Neural Networks for Classification of Audio Signals." arXiv preprint. arxiv.org/...1807.03418
- ^Kapishnikov, A., Venugopalan, S., Avci, B., Wedin, B., Terry, M., & Bolukbasi, T. (2021). "Guided Integrated Gradients: An Adaptive Path Method for Removing Noise." *CVPR 2021*. arxiv.org/...2106.09788
- ^1 ^2Smilkov, D., Thorat, N., Kim, B., Viegas, F., & Wattenberg, M. (2017). "SmoothGrad: Removing Noise by Adding Noise." arXiv preprint. arxiv.org/...1706.03825
- ^Janizek, J. D., Sturmfels, P., & Lee, S. I. (2021). "Explaining Explanations: Axiomatic Feature Interactions for Deep Networks." *Journal of Machine Learning Research*. arxiv.org/...2002.04138
- ^Kapishnikov, A., Bolukbasi, T., Viegas, F., & Terry, M. (2019). "XRAI: Better Attributions Through Regions." *ICCV 2019*. arxiv.org/...1906.02825
- ^1 ^2Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). "Sanity Checks for Saliency Maps." *NeurIPS 2018*. arxiv.org/...1810.03292
- ^Ghorbani, A., Abid, A., & Zou, J. (2019). "Interpretation of Neural Networks is Fragile." *AAAI 2019*. arxiv.org/...1710.10547
- ^Simonyan, K., Vedaldi, A., & Zisserman, A. (2014). "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps." *ICLR Workshop*. arxiv.org/...1312.6034
- ^Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why Should I Trust You? Explaining the Predictions of Any Classifier." *KDD 2016*. arxiv.org/...1602.04938
- ^Captum library on GitHub. github.com/...captum
- ^PAIR-code Saliency library on GitHub. github.com/...saliency
- ^Alibi library documentation. "Integrated Gradients." Seldon. docs.seldon.io/...IntegratedGradients
- ^InterpretML on GitHub. github.com/...interpret
- Molnar, C. (2022). *Interpretable Machine Learning: A Guide for Making Black Box Models Explainable*. Chapter on gradient-based attribution. christophm.github.io/interpretable-ml-book
- Ancona, M., Ceolini, E., Oztireli, C., & Gross, M. (2018). "Towards Better Understanding of Gradient-Based Attribution Methods for Deep Neural Networks." *ICLR 2018*. arxiv.org/...1711.06104
- ^1 ^2 ^3 ^4SHAP documentation. "shap.GradientExplainer." shap.readthedocs.io/...shap.GradientExplainer
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
5 revisions · v5 · 4,939 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check of the full article against its cited sources (110 claims, 24 sources fetched); 14 material + 14 minor fixed
Cite this page: AI Wiki. "Integrated Gradients." aiwiki.ai, updated 25 Jun 2026, fact-checked 5 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/integrated_gradients