Citation and evidence

Fisher information

20 min full readUpdated 21 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

Fisher information is a measure, defined by the statistician Ronald A. Fisher, of how much information an observable random variable carries about an unknown parameter of the probability distribution that generates it. Formally it is the expected squared derivative (for several parameters, the expected outer product of the gradient) of the log-likelihood with respect to the parameter, and under mild regularity conditions it equals the negative expected second derivative of the log-likelihood [1]. A distribution whose likelihood changes sharply when the parameter moves has high Fisher information, so the data pin the parameter down precisely; a flat likelihood carries little information [1].

The quantity runs through classical statistics. It sets the large-sample variance of the maximum likelihood estimator, gives the Cramér-Rao lower bound on the variance of unbiased estimators, defines Jeffreys's reparameterization-invariant prior in Bayesian statistics, and measures model complexity in the minimum description length approach to model selection [1]. In machine learning, the Fisher information matrix of a neural network's predictive distribution is the curvature matrix behind natural gradient descent and its approximations such as K-FAC, the trust-region constraint in Trust Region Policy Optimization, the parameter-importance weights of elastic weight consolidation in continual learning, and a range of methods for model merging, quantization and unlearning [2][3][15][4][5][6][7].

Definition

Single parameter

Let $f(x \mid \theta)$ be a probability density or mass function for a random variable $X$ with a scalar parameter $\theta$. The derivative of the log-likelihood, $\frac{d}{d\theta}\log f(x \mid \theta)$, is called the score function; it describes how sensitive the model is to a change in $\theta$ at a particular outcome $x$ [1]. The Fisher information is the expected square of the score, taken under the model itself:

IX(θ)=EX∼f(⋅∣θ)[(ddθlog⁡f(X∣θ))2]I_X(\theta) = \mathbb{E}_{X \sim f(\cdot \mid \theta)}\left[\left(\frac{d}{d\theta}\log f(X \mid \theta)\right)^2\right]

For a discrete variable the expectation is a sum over outcomes weighted by $f(x \mid \theta)$, and for a continuous variable it is an integral [1]. Under mild regularity conditions an equivalent definition is

IX(θ)=−E[d2dθ2log⁡f(X∣θ)],I_X(\theta) = -\mathbb{E}\left[\frac{d^2}{d\theta^2}\log f(X \mid \theta)\right],

so Fisher information is also the expected curvature of the log-likelihood [1]. Under the same conditions the score has mean zero, which makes $I(\theta)$ the variance of the score [8]. The conditions Ly and colleagues list for a regular parametric model include an open parameter space, a smoothness requirement on how the distribution depends on the parameter, and a non-singular information matrix [1].

Fisher information matrix

When $\theta$ is a vector of $d$ parameters, the score is a gradient vector and the Fisher information becomes a $d \times d$ matrix, usually written $F$ or $I(\theta)$:

F=E[∇θlog⁡p(x∣θ) ∇θlog⁡p(x∣θ)⊤]=−E[∇θ2log⁡p(x∣θ)]F = \mathbb{E}\left[\nabla_\theta \log p(x \mid \theta)\,\nabla_\theta \log p(x \mid \theta)^\top\right] = -\mathbb{E}\left[\nabla_\theta^2 \log p(x \mid \theta)\right]

The first form shows that $F$ is positive semi-definite, since it is an expectation of outer products; the second shows that it is the negative expected Hessian of the log-likelihood [2]. Off-diagonal entries are generally non-zero; when the $(i, j)$ entry is zero, the parameters $\theta_i$ and $\theta_j$ are called orthogonal [1].

For a conditional model $p(y \mid x, \theta)$, such as a classifier or regression network, the Fisher is the expectation over inputs of the Fisher of the predictive distribution $p(y \mid x, \theta)$, with the labels $y$ drawn from the model rather than from the training data [2]. In practice the input distribution is replaced by the training inputs [2].

Worked examples

ModelParameterFisher information per observation
Bernoulli (coin flip)probability of heads $\theta$$1/(\theta(1-\theta))$
Normal, variance $\sigma^2$ knownmean $\theta$$1/\sigma^2$
Normal$(\mu, \sigma)$diagonal matrix with entries $1/\sigma^2$ and $2/\sigma^2$
Laplace with scale $b$location $\theta$$1/b^2$

Expanded article table

All four results are worked out in the tutorial by Ly and colleagues [1]. The Bernoulli case shows that Fisher information can depend on the parameter: it goes to infinity as $\theta$ approaches 0 or 1, and $\theta = 1/2$ is the least informative case [1].

"The Fisher Information", video from Mutual Information, published May 5, 2021. Watch on YouTube.

Key properties

  • Additivity. For $n$ independent and identically distributed observations, the information in the whole sample is $n$ times the information in one observation, $I_{X^n}(\theta) = n I_X(\theta)$. For this reason $I_X(\theta)$ is also called the unit Fisher information [1]. The same additivity holds for the matrix version [1].
  • Information cannot be created by summarizing. For any statistic $T$ computed from the sample, $I_{X^n}(\theta) \geq I_T(\theta)$, with equality if and only if $T$ is a sufficient statistic for $\theta$. In a sequence of coin flips, the number of heads carries exactly as much information as the full sequence [1].
  • Link to KL divergence. For a small parameter change $d$, the Kullback-Leibler divergence between the model at $\theta + d$ and at $\theta$ is $\tfrac{1}{2} d^\top F d$ plus terms of third and higher order in $d$. The Fisher matrix is therefore the local quadratic approximation of KL divergence around $\theta$ [2].
  • Riemannian metric. Used as a metric tensor that varies smoothly over parameter space, the Fisher matrix turns a family of distributions into a Riemannian manifold [2]. In honor of C. R. Rao's 1945 contribution, this metric is often called the Fisher-Rao metric [1]. The study of this geometry is known as information geometry, and Shun-ichi Amari used it to derive learning algorithms [2][9].

History

Fisher's 1922 paper "On the Mathematical Foundations of Theoretical Statistics" defined consistency, efficiency and sufficiency, and, as the statistician David Hand notes in a 2015 commentary, those definitions made a concept of information relevant to statistical analysis [10]. In that paper Fisher defined the "intrinsic accuracy" of an error curve as the weight in large samples, divided by the number in the sample, of the statistic of location that satisfies the criterion of sufficiency [11]. The same paper criticized the use of a uniform prior on the grounds that it implies a non-uniform distribution for transformed versions of the parameter, which might equally well have been chosen to parameterize the problem [10]. Jeffreys later rejected the uniform prior as a default for the same reason, its dependence on the parameterization [1].

In "Theory of Statistical Estimation" (1925), Fisher built his account of efficiency on the score function. Writing the score as $X = \partial \log \phi / \partial \theta$ for the likelihood $\phi$ of a sample, he worked with the quantity now called the Fisher information in the sample, noted that it could be computed either from the expected squared first derivative or from the negative expected second derivative of the log-likelihood, and took its reciprocal, $1/I(\theta)$, as the large-sample variance of the maximum likelihood estimator [8]. His "analysis of variance" argument in that paper, which treated the estimate's asymptotic normal distribution as exact, concluded that any consistent, asymptotically normal estimate has variance $V \geq 1/I(\theta)$, the result now called the information inequality [8]. According to the historian of statistics Stephen Stigler, Arthur Bowley pointed out in 1935 that work by Edgeworth from 1908 and 1909 bore at least a superficial similarity to this inequality [8].

Fisher's proofs left gaps, and several mathematicians, including Joseph Doob, Abraham Wald and Harald Cramér, later worked to supply rigorous support for the theory [8]. A finite-sample version of the bound for unbiased estimators was published by Maurice Fréchet (1943), C. R. Rao (1945, in the Bulletin of the Calcutta Mathematical Society) and Cramér (1946), and is known as the Cramér-Rao or Cramér-Fréchet-Rao bound [1]. Harold Jeffreys proposed his invariant prior based on Fisher information in 1946 [1].

Uses in statistical inference

Cramér-Rao bound

The Cramér-Rao bound states that the variance of an unbiased estimator $T_n$ of $\theta$ from $n$ independent observations cannot be lower than the inverse Fisher information: $n \operatorname{Var}(T_n) \geq I_X^{-1}(\theta)$ [1]. Unlike the asymptotic results below, the bound also holds for finite $n$ [1]. Its reach is limited because the class of unbiased estimators is restrictive and in general does not include the maximum likelihood estimator [1].

Asymptotic normality of the maximum likelihood estimator

For independent, identically distributed data from a regular model, the rescaled error of the maximum likelihood estimator converges in distribution to a normal distribution whose variance is the inverse Fisher information:

n (θ^−θ∗)→DN(0,IX−1(θ∗))\sqrt{n}\,(\hat\theta - \theta^*) \xrightarrow{D} \mathcal{N}\left(0, I_X^{-1}(\theta^*)\right)

where $\theta^$ is the true parameter value. For large $n$, the standard error of the estimator is approximately $1/\sqrt{n I_X(\theta^)}$ [1]. This approximation is the basis for confidence intervals and hypothesis tests, and for choosing sample sizes. Ly and colleagues give the example of a coin-flip experiment: to make the standard error at most 0.1 in the least informative case $\theta = 1/2$, one needs $n = 25$ flips [1]. The approximation can perform poorly when $n$ is small, and it assumes the model is correctly specified [1]. In the vector case the inverse Fisher information matrix gives the approximate covariance of the estimator [1].

Observed and expected information

The quantity defined above is sometimes called the expected Fisher information, to distinguish it from the observed information, the negative second derivative of the log-likelihood evaluated at the maximum likelihood estimate for the data actually observed. In a 1978 Biometrika paper, Bradley Efron and David Hinkley gave a frequentist justification for using the observed information when estimating the variance of the maximum likelihood estimator in one-parameter families, showing that the reciprocal of the observed information approximates the estimator's conditional variance given an appropriate ancillary statistic [12].

Jeffreys prior

A uniform prior on a parameter is not invariant: a different parameterization of the same model gives a different posterior and possibly different conclusions [1]. Jeffreys (1946) proposed instead the prior

gJ(θ)=1VIX(θ),V=∫ΘIX(θ) dθ,g_J(\theta) = \frac{1}{V}\sqrt{I_X(\theta)}, \qquad V = \int_\Theta \sqrt{I_X(\theta)}\,d\theta,

which gives the same posterior however the model is parameterized [1]. In the multiparameter case the prior uses the square root of the determinant of the Fisher information matrix [1].

Model complexity

In the minimum description length framework, the Fisher information approximation (FIA) to a model's code length adds two complexity terms to the negative log-likelihood: one that grows with the number of parameters and the logarithm of the sample size, and one equal to the logarithm of $\int \sqrt{\det I(\theta)},d\theta$ over the parameter space. The integral in the second term is the normalizing constant of the Jeffreys prior, which Ly and colleagues interpret as the volume of the model in the space of distributions: a model with a larger volume has more capacity to fit data and receives a larger penalty [1].

Uses in machine learning

Natural gradient descent

Ordinary gradient descent follows the steepest-descent direction measured with Euclidean distance in parameter space, so its behavior depends on how the model happens to be parameterized. Shun-ichi Amari's 1998 paper "Natural Gradient Works Efficiently in Learning" argued that when the parameter space has an underlying geometric structure, the ordinary gradient does not point in the steepest direction, while the natural gradient does [9]. The natural gradient of an objective $h$ is the ordinary gradient premultiplied by the inverse Fisher matrix [2]:

∇~h=F−1∇h\tilde\nabla h = F^{-1} \nabla h

Because the Fisher matrix is the local quadratic approximation of KL divergence, the negative natural gradient is the steepest-descent direction in the space of distributions when distance is measured locally by KL divergence, and the direction it defines is invariant to smooth, invertible reparameterizations of the model [2]. Amari also showed that natural gradient online learning is Fisher efficient, meaning it asymptotically performs as well as optimal batch estimation, and suggested that it may ease the plateau problem of backpropagation in multilayer perceptrons [9].

James Martens' 2020 analysis in the Journal of Machine Learning Research recasts natural gradient descent as a second-order optimization method in which the Fisher replaces the Hessian [2]. For a model trained with negative log-likelihood loss, the Fisher is the expected Hessian of the loss under the model's own predictive distribution, while the true Hessian takes the expectation under the training targets, so the two approximate each other [2]. When the predictive distribution is an exponential family with natural parameters given by the network output (as with squared error or softmax cross-entropy), the Fisher equals the generalized Gauss-Newton matrix [2]. Martens also points out that the parameterization invariance holds exactly only for an idealized method taking infinitesimally small steps, and that the second-order view motivates update damping, such as trust regions and Tikhonov damping, because large discrete steps can leave the region where the local quadratic approximation is accurate [2].

Approximations for large networks: K-FAC

For a network with millions of parameters, storing and inverting the full Fisher matrix is impractical, since the matrix needs memory quadratic in the number of parameters [2][5]. K-FAC (Kronecker-factored Approximate Curvature), introduced by James Martens and Roger Grosse at ICML 2015, approximates large blocks of the Fisher, one per layer, as the Kronecker product of two much smaller matrices [3]. The result is neither diagonal nor low-rank but can be inverted efficiently. The authors reported that K-FAC costs only several times more than a plain stochastic gradient per step while making much more progress per update, and that, because the cost of storing and inverting the approximation does not depend on the amount of data used to estimate it, the method works well in highly stochastic regimes [3]. The optimizer article covers K-FAC and related curvature-based methods in more detail.

Earlier, Razvan Pascanu and Yoshua Bengio related natural gradient descent to Hessian-free optimization, Krylov subspace descent and TONGA, and studied it for training deep models [13].

Reinforcement learning: natural policy gradient and TRPO

Sham Kakade's "A Natural Policy Gradient" (NIPS 2001) applied the natural gradient to policy gradient methods in reinforcement learning [14]. For each state $s$, the policy $\pi(a; s, \theta)$ is a distribution over actions with its own Fisher information matrix, and Kakade used the average of these matrices under the policy's stationary state distribution as the metric [14]. He reported that the natural gradient moves toward choosing a greedy optimal action rather than just a better one, and showed large performance gains on simple Markov decision processes and on Tetris [14].

Trust Region Policy Optimization (TRPO), by John Schulman and colleagues (ICML 2015), constrains each policy update by the average KL divergence between the old and new policies [15]. The authors show that the natural policy gradient is a special case of their update, obtained with a linear approximation to the objective and a quadratic approximation to the KL constraint whose matrix is the Fisher [15]. TRPO solves for the search direction $F^{-1}g$ with the conjugate gradient algorithm, which needs only Fisher-vector products, and its authors computed the Fisher analytically as the Hessian of the KL divergence rather than as the covariance matrix of the gradients [15].

The empirical Fisher and adaptive optimizers

A widely used shortcut, often called the empirical Fisher, replaces the model's own predictions with the training labels, averaging the outer products of per-example gradients of the training loss [2][16]. It is essentially free to compute once gradients are available, but it is not equivalent to the Fisher in general, and Martens notes that it is often incorrectly called the Fisher or the Gauss-Newton matrix [2]. The true Fisher can instead be estimated by sampling labels from the model's predictive distribution [2].

The original Adam paper (Kingma and Ba, ICLR 2015) described its second-moment estimate as "an approximation to the diagonal of the Fisher information matrix" and noted that Adam preconditions more conservatively than natural gradient descent by using the square root of the inverse of that diagonal [17]. Frederik Kunstner, Lukas Balles and Philipp Hennig disputed this line of argument in 2019, arguing that the empirical Fisher, unlike the Fisher, does not generally capture second-order information, that the conditions under which it approaches the Fisher are unlikely to hold in practice, and that it can distort the update even on simple linear regression problems [16]. They suggested that empirical-Fisher methods may succeed by adapting to gradient noise rather than to curvature, and observed that methods originally defined with the Fisher, such as elastic weight consolidation and K-FAC, have been reimplemented by third parties with the empirical Fisher [16].

Continual learning: elastic weight consolidation

Elastic weight consolidation (EWC), published by James Kirkpatrick and colleagues at DeepMind in the Proceedings of the National Academy of Sciences in 2017, uses Fisher information to reduce catastrophic forgetting when a network learns tasks in sequence [4][18]. Following David MacKay's work on the Laplace approximation, EWC approximates the posterior over parameters after task A as a Gaussian centered on the learned parameters $\theta^*_A$, with a diagonal precision given by the diagonal of the Fisher information matrix [4]. Training on task B then minimizes

L(θ)=LB(θ)+∑iλ2Fi(θi−θA,i∗)2\mathcal{L}(\theta) = \mathcal{L}_B(\theta) + \sum_i \frac{\lambda}{2} F_i \left(\theta_i - \theta^*_{A,i}\right)^2

where $F_i$ is the Fisher information of parameter $i$ and $\lambda$ sets the importance of the old task [4]. The authors justified the choice of the Fisher with three properties: it is equivalent to the second derivative of the loss near a minimum, it can be computed from first-order derivatives alone, and it is guaranteed to be positive semi-definite [4]. EWC was tested on permuted MNIST classification tasks and on learning several Atari 2600 games in sequence, and its run time is linear in the number of parameters and training examples [4]. The authors acknowledged that using a point estimate of the posterior's variance, as in a Laplace approximation, is a significant weakness [4].

Model merging, quantization and unlearning

The same Laplace-approximation reasoning has been applied to combining models. In "Fisher merging", Michael Matena and Colin Raffel approximate each fine-tuned model's posterior as a Gaussian whose precision is its diagonal Fisher information, then pick the parameters that are most probable under all the posteriors together [5]. The solution is a per-parameter weighted average,

θ∗(j)=∑i=1MλiFi(j)θi(j)∑i=1MλiFi(j),\theta^{*(j)} = \frac{\sum_{i=1}^{M} \lambda_i F_i^{(j)} \theta_i^{(j)}}{\sum_{i=1}^{M} \lambda_i F_i^{(j)}},

in which each model's value of parameter $j$ is weighted by its Fisher information and a merging coefficient $\lambda_i$ [5]. Because the full Fisher needs memory quadratic in the number of parameters, they used the diagonal, estimated from per-example gradients with the expectation over labels taken under the model's predictive distribution [5]. Later work extends the idea to large language models: a March 2026 preprint describing FIM-Merging uses a diagonal Fisher computed from random token inputs to set per-layer merging coefficients when combining a base model with a long chain-of-thought reasoning model [19].

In post-training quantization, SqueezeLLM (Sehoon Kim and colleagues, ICML 2024) needs to know how sensitive the loss is to perturbing each weight. Because the Hessian is costly, it approximates the Hessian by a Fisher matrix built from gradients over a sample dataset, simplifies that to a diagonal, and uses the diagonal entries as weights in a k-means clustering that places quantization levels near sensitive weights [6]. In machine unlearning for language models, methods such as FILA, and VILA, proposed in 2025 as an improvement on it, use Fisher information to identify the parameters most associated with the data to be forgotten; the VILA authors argued that FILA did not adequately account for the assumptions underlying Fisher information [7].

Fisher score features

Fisher information also underlies ways of turning a generative model into features. Tommi Jaakkola and David Haussler (NIPS 1998) called the gradient of the log-likelihood of an example the Fisher score and defined the Fisher kernel as the inner product of these score vectors relative to the local Riemannian metric given by the Fisher information matrix, so that kernel methods such as support vector machines can use probabilistic models like hidden Markov models [20]. They noted that the information matrix itself is asymptotically immaterial and that a simpler kernel using only the scores can substitute for it [20]. In computer vision, Task2Vec (Alessandro Achille and colleagues, 2019) represents a whole classification task by an embedding computed from estimates of the Fisher information matrix of a fixed "probe network", and uses these embeddings to predict task similarity and to choose pre-trained feature extractors [21].

Limitations

  • Regularity. The equivalence of the two definitions, asymptotic normality of the maximum likelihood estimator and the Cramér-Rao bound all depend on regularity conditions, and the large-sample approximations can perform poorly at small sample sizes or under model misspecification [1].
  • Scale. The full Fisher matrix of a modern network is too large to store or invert, so machine-learning methods rely on diagonal, block-diagonal, Kronecker-factored or matrix-free approximations [2][3][5].
  • Empirical versus true Fisher. Many implementations use the empirical Fisher, which can behave very differently from the Fisher and lacks its connection to the Hessian [2][16].
  • Locality. The Fisher describes the model only near the current parameters. Natural gradient invariance holds only for small steps [2], EWC relies on a point estimate of posterior variance [4], and Fisher merging may work less well for models whose parameters are far apart, which is why its authors restricted it to models trained from the same initialization [5].

Continue reading

More from See also

More to read

Topics:Machine LearningMathematicsStatisticsTraining & Optimization

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul Grasman and Eric-Jan Wagenmakers. "A Tutorial on Fisher Information". Journal of Mathematical Psychology 80: 40-55, 2017 (arXiv:1705.01064). arxiv.org/...1705.01064
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20James Martens. "New Insights and Perspectives on the Natural Gradient Method". Journal of Machine Learning Research 21: 1-76, 2020. jmlr.org/...17-678
  3. ^1 ^2 ^3 ^4James Martens and Roger Grosse. "Optimizing Neural Networks with Kronecker-factored Approximate Curvature". Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015 (arXiv:1503.05671). arxiv.org/...1503.05671
  4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran and Raia Hadsell. "Overcoming catastrophic forgetting in neural networks". arXiv:1612.00796, 2016. arxiv.org/...1612.00796
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7Michael Matena and Colin Raffel. "Merging Models with Fisher-Weighted Averaging". arXiv:2111.09832, 2021. arxiv.org/...2111.09832
  6. ^1 ^2Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney and Kurt Keutzer. "SqueezeLLM: Dense-and-Sparse Quantization". ICML 2024 (arXiv:2306.07629). arxiv.org/...2306.07629
  7. ^1 ^2Yejin Kim, Eunwon Kim, Buru Chang and Junsuk Choe. "Improving Fisher Information Estimation and Efficiency for LoRA-based LLM Unlearning". Conference on Language Modeling (COLM), 2025 (arXiv:2508.21300, August 2025). arxiv.org/...2508.21300
  8. ^1 ^2 ^3 ^4 ^5Stephen M. Stigler. "The Epic Story of Maximum Likelihood". Statistical Science 22(4): 598-620, 2007 (arXiv:0804.2996). arxiv.org/...0804.2996
  9. ^1 ^2 ^3Shun-ichi Amari. "Natural Gradient Works Efficiently in Learning". Neural Computation 10(2): 251-276, 1998. mlanthology.org/...amari1998neco-natural
  10. ^1 ^2David J. Hand. "From evidence to understanding: a commentary on Fisher (1922) 'On the mathematical foundations of theoretical statistics'". Philosophical Transactions of the Royal Society A 373(2039): 20140252, 2015. europepmc.org/...PMC4360088
  11. ^R. A. Fisher. "On the Mathematical Foundations of Theoretical Statistics". Philosophical Transactions of the Royal Society of London, Series A 222: 309-368, 1922. doi.org/...rsta.1922.0009
  12. ^Bradley Efron and David V. Hinkley. "Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information". Biometrika 65(3): 457-483, December 1978. academic.oup.com/...233667
  13. ^Razvan Pascanu and Yoshua Bengio. "Revisiting Natural Gradient for Deep Networks". arXiv:1301.3584, 2013. arxiv.org/...1301.3584
  14. ^1 ^2 ^3Sham M. Kakade. "A Natural Policy Gradient". Advances in Neural Information Processing Systems 14 (NIPS 2001). proceedings.neurips.cc/...94c56c69108433e-Abstract
  15. ^1 ^2 ^3 ^4John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan and Pieter Abbeel. "Trust Region Policy Optimization". ICML 2015 (arXiv:1502.05477). arxiv.org/...1502.05477
  16. ^1 ^2 ^3 ^4Frederik Kunstner, Lukas Balles and Philipp Hennig. "Limitations of the Empirical Fisher Approximation for Natural Gradient Descent". arXiv:1905.12558, 2019. arxiv.org/...1905.12558
  17. ^Diederik P. Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization". ICLR 2015 (arXiv:1412.6980). arxiv.org/...1412.6980
  18. ^James Kirkpatrick et al. "Overcoming catastrophic forgetting in neural networks". Proceedings of the National Academy of Sciences 114(13): 3521-3526, 2017. doi.org/...pnas.1611835114
  19. ^Tian Xia. "Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMs". arXiv:2603.21705, March 2026. arxiv.org/...2603.21705
  20. ^1 ^2Tommi Jaakkola and David Haussler. "Exploiting Generative Models in Discriminative Classifiers". Advances in Neural Information Processing Systems 11 (NIPS 1998). proceedings.neurips.cc/...c8b88e879465a1e-Abstract
  21. ^Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto and Pietro Perona. "Task2Vec: Task Embedding for Meta-Learning". arXiv:1902.03545, 2019. arxiv.org/...1902.03545

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

2 revisions · v2 · 4,013 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 92 claims checked against 22 cited and independent sources; 0 material and 6 minor corrections applied before publication.

Cite this page: AI Wiki. "Fisher information." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/fisher_information

Edit article