Decision theory
Decision theory is the study of how an agent should choose, or does choose, among actions whose consequences are uncertain. In the formulation of philosophers Katie Steele and H. Orri Stefánsson, it is concerned with the reasoning behind an agent's choices and is as much a theory of beliefs, desires and other "preference attitudes" as it is a theory of choice [1]. Its orthodox normative version, expected utility theory, says that under uncertainty an agent should prefer the option with the greatest expected desirability or value [1]. The field draws on probability theory, economics, statistics and philosophy, and its landmarks include Frank Ramsey's early proposal for measuring belief and desire, the expected utility theorems of John von Neumann and Oskar Morgenstern and of Leonard Savage, Richard Jeffrey's theory of decision, the paradoxes of Maurice Allais and Daniel Ellsberg, and the dispute between causal and evidential decision theory over Newcomb's problem [1][2][3].
Decision theory is one of the mathematical foundations of artificial intelligence. Stuart Russell and Peter Norvig's textbook Artificial Intelligence: A Modern Approach defines a rational agent as one that "acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome," and states that the modern notion of rational decision making under uncertainty involves maximizing expected utility [5]. The same framework underlies Markov decision processes and reinforcement learning, the decision step of probabilistic classification, and several debates in AI safety about how advanced AI systems will behave and how they should reason [5][7][15][19][27].
Normative and descriptive decision theory
Decision theory is usually divided into normative theories, which say how agents ought to decide, and descriptive or predictive theories, which say how people actually decide [2]. R. A. Briggs notes that classical economics often uses expected utility theory descriptively or predictively, that the theory makes faulty predictions about people's decisions in many real choice situations, and that this failure does not settle whether people should decide on the basis of expected utility [2]. Steele and Stefánsson describe the normative project as a "minimal account of rationality": it asks what criteria an agent's preferences should satisfy in any circumstances, while setting aside more substantial questions about which desires are appropriate and which beliefs are reasonable [1].
The descriptive side became a research program in its own right. Herbert Simon received the 1978 economics prize "for his pioneering research into the decision-making process within economic organizations"; his work held that people deviate from strict rationality when making choices [9]. Russell and Norvig describe Simon's contribution as showing that models based on satisficing, making decisions that are "good enough" rather than calculating an optimal one, described actual human behavior better [5]. Daniel Kahneman shared the 2002 prize "for having integrated insights from psychological research into economic science, especially concerning human judgment and decision-making under uncertainty" [8]. The Royal Swedish Academy of Sciences said Kahneman had shown how human decisions may systematically depart from those predicted by standard economic theory, and that with Amos Tversky he formulated prospect theory as an alternative that better accounts for observed behavior [8]. Their paper "Prospect Theory: An Analysis of Decision under Risk" appeared in Econometrica in 1979 [10].
History
| Year | Contribution | Source |
|---|---|---|
| 1662 | Antoine Arnauld proposes a formula for maximizing the expected monetary value of a gamble | [5] |
| 1713 | Nicolaus Bernoulli poses the St. Petersburg problem in a letter to Pierre Rémond de Montmort | [4] |
| 1728 | Cramér, in a letter to Nicolaus Bernoulli, values the game by a square-root utility of money | [4] |
| 1738 | Daniel Bernoulli proposes maximizing expected utility, with utility the logarithm of wealth | [4][5] |
| 1926 | Frank Ramsey's "Truth and Probability" proposes a lottery-based measure of desirability | [1] |
| 1944 | John von Neumann and Oskar Morgenstern, Theory of Games and Economic Behavior, with the expected utility theorem | [1][5] |
| 1953 | Maurice Allais publishes his counterexample to the independence axiom in Econometrica | [1][11] |
| 1954 | Leonard Savage, The Foundations of Statistics, subjective expected utility | [1] |
| 1957 | Richard Bellman formalizes Markov decision processes | [5] |
| 1961 | Daniel Ellsberg, "Risk, Ambiguity, and the Savage Axioms" | [2][12] |
| 1965 | Richard Jeffrey, The Logic of Decision | [1] |
| 1969 | Robert Nozick publishes Newcomb's problem | [3][14] |
| 1978 | Allan Gibbard and William Harper distinguish causal from evidential decision theory | [3] |
| 1979 | Daniel Kahneman and Amos Tversky, prospect theory | [8][10] |
| 1999 | James Joyce's general representation theorem for causal and evidential decision theory | [3] |
| 2017 | Eliezer Yudkowsky and Nate Soares, functional decision theory | [27] |
From expected value to expected utility
The first mathematical analyses of decisions under uncertainty dealt with the monetary value of bets. Russell and Norvig trace the maximum-expected-value formula to Antoine Arnauld in 1662 [5]. The St. Petersburg paradox showed why expected monetary value is not enough. Nicolaus Bernoulli introduced it in 1713 [4]. In the standard version a fair coin is tossed until it first lands heads, and the player wins $2^n$ dollars if that happens on the $n$-th toss, so the expected monetary value of the game is
yet, as Martin Peterson puts it, claiming that a rational agent should pay millions or even billions to play seems absurd [4]. In a 1728 letter to Nicolaus Bernoulli, the mathematician Cramér suggested that the game be valued by the utility of money rather than money itself, and Daniel Bernoulli made a similar proposal in a 1738 paper in the journal of the Imperial Academy of Sciences in St. Petersburg, arguing that the utility of wealth equals its logarithm, so that large but improbable prizes contribute less [4]. Bernoulli acknowledged in the final text that Cramér had reached nearly the same idea independently [4]. An English translation of Bernoulli's paper, "Exposition of a New Theory on the Measurement of Risk", was published in Econometrica in 1954 [13]. Russell and Norvig summarize Bernoulli's idea as the proposal that the marginal utility of an additional quantity of money diminishes as one acquires more money [5].
Axiomatic foundations
Ramsey proposed in 1926 that the strength of a preference could be measured by asking what chance of a better prize in a lottery an agent would accept in place of a sure option, and gave axioms that yield roughly the same result as later theories, though without a full proof [1]. Von Neumann and Morgenstern's 1944 book gave axioms for preferences over lotteries and proved that preferences satisfying them can be represented as maximizing expected utility [1]. Savage's The Foundations of Statistics (1954) extended the approach to choices whose outcomes depend on uncertain states of the world rather than on given probabilities; Steele and Stefánsson call it the best-known normative theory of choice under uncertainty, in particular within economics and the decision sciences [1]. Jeffrey's The Logic of Decision (1965) made propositions the objects of both belief and desire, and the mathematician Ethan Bolker proved a representation theorem for Jeffrey's framework in 1966 and 1967 [1].
Expected utility theory
Utility and lotteries
In von Neumann and Morgenstern's setting an agent chooses among lotteries. A lottery $L$ yields outcome $O_k$ with probability $p_k$, and its expected utility is
where $u$ assigns a real number to each outcome [1]. The agent can be represented as an expected utility maximizer if, for any two lotteries, it weakly prefers $L_j$ to $L_i$ exactly when $EU(L_i) \le EU(L_j)$ [1]. The theorem identifies four conditions on preferences under which such a representation exists [1]:
| Axiom | Informal content |
|---|---|
| Completeness | Any two options can be compared |
| Transitivity | If $A$ is weakly preferred to $B$ and $B$ to $C$, then $A$ is weakly preferred to $C$ |
| Continuity | If $A \preceq B \preceq C$, some probability mixture of $A$ and $C$ is exactly as good as $B$ |
| Independence | Mixing two lotteries with the same third lottery, in the same proportion, does not reverse the preference between them |
Preferences over a set of lotteries closed under probability mixture satisfy these axioms if and only if there is a utility function over outcomes, unique up to positive linear transformation, relative to which the preferences maximize expected utility [1]. Uniqueness up to positive linear transformation means that if $u$ represents the preferences, so does $a,u + b$ for any $a > 0$; utility in this sense has no absolute zero or unit, and it does not by itself support comparisons of utility between different people [1].
Subjective expected utility
Most real choices do not come with known probabilities, a distinction economists describe as choice under risk versus choice under uncertainty [1]. Savage's theory takes states of the world, outcomes and acts as its primitives. States are features of the world the agent does not control; acts are functions from states to outcomes; and the agent has preferences over all such acts [1]. Savage's axioms, including the "sure-thing principle," guarantee a probability function $P$ over states and a utility function $u$ over outcomes, with $u$ unique up to positive linear transformation, such that the agent prefers acts with higher
(written here for finitely many states) [1]. The probability function is interpreted as the agent's degrees of belief and the utility function as its desires [1]. Some writers call subjective expected utility theory Bayesian decision theory, a label that stresses its commitment to representing degrees of belief as probabilities [1].
Arguments for expected utility
Several arguments support the axioms. The money-pump argument defends transitivity: an agent with cyclic preferences, preferring $A$ to $C$ while being willing to trade $A$ for $B$ and $B$ for $C$, can be led through a series of trades, each acceptable by its own lights, that returns it to $A$ with less money than it started with [1]. Peter Hammond's dynamic consistency arguments hold that only an agent with expected utility preferences that updates by conditionalization can always carry out, in a sequential decision tree, a plan it judged optimal at the start [1]. I. J. Good proved in 1967 that an expected utility maximizer will always seek free evidence that bears on its decision, a result that connects decision theory to the value of information [1].
Paradoxes and alternatives
Allais and Ellsberg
The French economist Maurice Allais introduced his paradox in the early 1950s [1]; the paper, "Le Comportement de l'Homme Rationnel devant le Risque: Critique des Postulats et Axiomes de l'Ecole Americaine", appeared in Econometrica in 1953 [11]. In a version given by Steele and Stefánsson, a ticket numbered 1 to 100 is drawn at random [1]:
| Lottery | Ticket 1 | Tickets 2-34 | Tickets 35-100 |
|---|---|---|---|
| $L_1$ | $0 | $2,500 | $2,400 |
| $L_2$ | $2,400 | $2,400 | $2,400 |
| $L_3$ | $0 | $2,500 | $0 |
| $L_4$ | $2,400 | $2,400 | $0 |
Many people prefer $L_2$ to $L_1$ but $L_3$ to $L_4$ [1]. A common rationale is that in the first choice a sure $2,400 is not worth giving up for a slightly better chance of a higher prize, while in the second choice the worst outcome is $0 either way [1]. Because the two pairs differ only in the last column, which is the same within each pair, the independence axiom requires the same preference in both pairs, so these choices cannot be represented as maximizing expected utility [1].
Ellsberg's 1961 paradox concerns ambiguity rather than risk [2][12]. An urn holds 30 red balls and 60 balls that are white or yellow in unknown proportions. It seems reasonable to prefer a bet on red to a bet on white, but a bet on "white or yellow" to a bet on "red or yellow"; this combination of preferences also violates independence, whatever probabilities the agent assigns [2].
Briggs identifies three main responses to the two paradoxes: defend expected utility theory and call the preferences irrational, as Savage and Howard Raiffa did; accept the preferences as rational and adopt a more permissive theory, as Lara Buchak does; or redescribe the outcomes (for example, to include feelings of disappointment or elation) so that the preferences no longer conflict with the theory [2]. Buchak's risk-weighted expected utility theory, published in 2013, replaces Savage's sure-thing principle with a weaker axiom and adds a risk function representing a person's willingness to trade chances of something good for risks of something bad [1].
Other challenges
Critics have also questioned completeness. Some philosophers argue that the evidence may not commit a rational agent to precise degrees of belief, and represent belief by a set of probability functions instead of one, a position called imprecise probabilism [1]. The use of expected utility for catastrophic risks has been challenged in climate policy, where some have proposed that something closer to the precautionary principle should guide evaluation [1]. A further, more recent line of work concerns unawareness: an agent may not have conceived of all the possibilities relevant to its decision, which Steele and Stefánsson describe as a challenge for all existing normative decision theories [1].
Causal and evidential decision theory
Expected utility needs the probability of each state given an act. Taking this to be the ordinary conditional probability $P(S \mid A)$ gives evidential decision theory (EDT), associated with Jeffrey's framework; taking it to be a causal notion, such as the probability of the subjunctive conditional "if $A$ were done, $S$ would obtain", gives causal decision theory (CDT) [3]. Schematically,
where $V$ is Gibbard and Harper's "value", which measures how good the news of an act would be, and $U$ is their "utility", which measures the act's efficacy in bringing about the agent's goals [3]. Paul Weirich summarizes the causal view in the slogan that a good decision aims to produce a good outcome rather than evidence of a good outcome [3].
The two theories diverge in Newcomb's problem, which Robert Nozick published in 1969 and named after the physicist William Newcomb, who first formulated it [3][14]. An agent may take an opaque box, or the opaque box together with a transparent box holding $1,000. The opaque box holds $1,000,000 if a reliable predictor predicted that the agent would take only the opaque box, and nothing if it predicted that the agent would take both [3]:
| Predicted one box | Predicted two boxes | |
|---|---|---|
| Take one box | $1,000,000 | $0 |
| Take two boxes | $1,001,000 | $1,000 |
Taking both boxes dominates: whatever was predicted, it yields $1,000 more. But because the prediction is reliable, expected utility computed with conditional probabilities favors taking one box [3]. Allan Gibbard and William Harper's 1978 paper elaborated a resolution proposed by Robert Stalnaker, under which expected utility maximization agrees with dominance, and distinguished causal from evidential decision theory; versions of causal decision theory were later given by Brian Skyrms and David Lewis [3]. The same structure appears in a prisoner's dilemma between psychological twins, where cooperating is evidence, but not a cause, that the twin also cooperates [3]. The most common objection to CDT is that it gives the wrong answer in Newcomb's problem, on the ground that one-boxers fare better than two-boxers; causal decision theorists reply that the problem is unusual in rewarding irrationality [3]. James Joyce's 1999 representation theorem yields either causal or evidential decision theory depending on how the probabilities in the expected utility formula are interpreted [3].
Decision theory in artificial intelligence
Rational agents and the standard model
Artificial Intelligence: A Modern Approach organizes the field around the idea of an intelligent agent that receives percepts and performs actions [6]. Russell and Norvig call the paradigm in which an agent optimizes an objective supplied to it the "standard model" and note that it also prevails in control theory, where a controller minimizes a cost function; in operations research, where a policy maximizes a sum of rewards; in statistics, where a decision rule minimizes a loss function; and in economics, where a decision maker maximizes utility [5]. They describe decision theory, which combines probability theory with utility theory, as a formal and complete framework for individual decisions under uncertainty, and contrast it with game theory, which handles interactions between agents and does not offer an unambiguous prescription for selecting actions [5]. They also note that work in economics and operations research had long developed apart from AI, partly because of the apparent complexity of making rational decisions, and that interest in decision-theoretic techniques for AI revived from the 1990s [5].
Because exactly optimal behavior is computationally infeasible in complex environments, the book adds the notion of limited rationality, acting appropriately when there is not enough time for all the computation one might like, while keeping perfect rationality as a starting point for analysis [5]. In the fourth edition (Pearson, 2021), "Making Simple Decisions" covers the basis of utility theory, constraints on rational preferences, the utility of money, multiattribute utility, decision networks and the value of information; "Making Complex Decisions" covers sequential decision problems, Markov decision processes, bandit problems and partially observable MDPs; and "Multiagent Decision Making" covers game theory [7].
Decision networks and influence diagrams
According to a 2005 retrospective by Ronald Howard and James Matheson, influence diagrams were invented in the mid-1970s and became a widely used tool for representing uncertain situations: a single diagram displays both the sequential and the informational structure of decisions, replacing awkward manipulation of decision trees [17]. Craig Boutilier wrote in 2005 that Howard and Matheson's article "Influence Diagrams" had a substantial impact on AI research as a model for decision making under uncertainty [18]. In a 1988 survey, Eric Horvitz, John Breese and Max Henrion argued that AI and the decision sciences share common roots, and that analyses of the limits of traditional AI reasoning techniques, together with more tractable decision-theoretic representations, had renewed interest in decision theory and decision analysis; the survey presented the belief network and influence diagram representations for automated reasoning [16]. Russell and Norvig's textbook devotes a section to decision networks and their evaluation [7].
Sequential decisions and reinforcement learning
Economists largely did not address decisions whose payoffs result from a sequence of actions. Russell and Norvig credit that topic to operations research and to Richard Bellman, whose 1957 work formalized the Markov decision process, which their book studies directly and again under the heading of reinforcement learning [5]. In the book's description of the standard model, operations research looks for a policy that maximizes a sum of rewards [5]. Its chapter on complex decisions covers utilities over time, optimal policies and the utilities of states, algorithms such as value iteration and policy iteration, multi-armed bandit problems, and partially observable MDPs [7].
Statistical decision theory in machine learning
In machine learning, decision theory usually appears as the step that turns probabilistic predictions into actions. Christopher Bishop's Pattern Recognition and Machine Learning (2006) separates an inference stage, in which training data are used to learn a model of the posterior class probabilities $p(C_k \mid \mathbf{x})$, from a decision stage that uses those probabilities to make optimal choices [15]. Bishop's example is cancer screening from an X-ray image, where diagnosing a healthy patient as ill and missing a cancer have very different consequences. He encodes these with a loss matrix $L_{kj}$, the cost of assigning an input whose true class is $C_k$ to class $C_j$ (in his illustration 1 for a false alarm and 1000 for a missed cancer), and minimizes the expected loss
which is achieved by assigning each new input to the class $j$ that minimizes $\sum_k L_{kj}, p(C_k \mid \mathbf{x})$ [15]. Bishop notes that a utility function can be used instead, since utility is simply the negative of the loss, and that a classifier can also be given a reject option that leaves the most uncertain cases to a human expert [15]. This loss-function formulation is the sense in which statistics, in Russell and Norvig's description, uses a decision rule that minimizes a loss function [5]. Related ideas appear in loss functions, empirical risk minimization and Bayesian inference.
Decision theory in AI safety
Expected utility maximizers and instrumental incentives
Decision-theoretic models of agents are central to arguments about risks from advanced AI. Stephen Omohundro's 2008 paper "The Basic AI Drives" argued that sufficiently advanced AI systems of almost any design would have "drives", tendencies present unless explicitly counteracted: self-improving systems would be driven to represent their goals as economic utility functions and to approximate rational economic behavior, which would lead almost all of them to protect their utility functions from modification, and systems would also tend to protect themselves from harm and to acquire resources [19]. Omohundro invoked the expected utility theorem in the form that a system can represent its preferences by the expectation of a utility function unless it has "vulnerabilities" that cause it to lose resources without benefit [19]. These arguments underlie the concept of instrumental convergence.
The 2015 paper "Corrigibility" by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky and Stuart Armstrong, presented at an AAAI workshop, starts from the observation that an agent maximizing the expectation of a utility function $U$ is by default incentivized to preserve its $U$-maximizing behavior and so to resist having its goals changed [20]. The paper calls an AI system "corrigible" if it cooperates with what its creators regard as a corrective intervention, and analyzes utility functions intended to make an agent shut down when a button is pressed without trying to prevent or cause the pressing [20]. Elliott Thornley's "The shutdown problem: an AI engineering puzzle for decision theorists", published in Philosophical Studies in 2024, proves three theorems suggesting that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of a shutdown button, even at a cost [23].
Uncertainty about objectives
Russell and Norvig argue that the standard model is probably not the right model in the long run, because it assumes that a fully specified objective can be supplied to the machine [5]. The fourth edition of Artificial Intelligence: A Modern Approach states that the authors previously defined the goal of AI as creating systems that maximize expected utility with the objective supplied by human designers, but no longer assume that the objective is fixed and known to the AI system; the system may be uncertain about the true objectives of the humans on whose behalf it acts [6]. The book argues that a machine that knows it does not know the complete objective has an incentive to act cautiously, ask permission, learn more about human preferences and defer to human control [5]. Two 2016 papers by Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell model this setting. "Cooperative Inverse Reinforcement Learning" defines the value alignment problem as a cooperative, partial-information game in which a human and a robot are both rewarded by the human's reward function, which the robot does not initially know [22]. "The Off-Switch Game" finds that an agent that takes its reward function for granted has an incentive to disable its off switch, except when the human is perfectly rational, and that an agent wants to preserve its off switch when it is uncertain about the utility of the outcome and treats the human's actions as evidence about it [21].
Coherence arguments
A recurring argument in AI alignment discussions holds that a sufficiently capable system will behave as an expected utility maximizer because any agent whose preferences violate the axioms can be exploited, as in the money pump. A 2021 AI Impacts summary by Katja Grace describes coherence arguments as holding that an entity whose preferences do not fit expected utility maximization can be manipulated into losing things it values [25]. In a 2018 post, Rohin Shah argued that coherence alone constrains behavior very little: given an agent's full policy, one can define a utility function over histories that assigns 1 to whatever the policy does and 0 otherwise, so that "no matter what the policy is, we can view the agent as an EU maximizer," and the coherence theorems therefore do not establish that an AI system must be goal-directed [24]. Grace's summary concludes that coherence does not logically imply goal-directed behavior, but that coherence arguments suggest some pressure for advanced AI to become more coherent, of unclear strength [25].
Newcomb-like problems and functional decision theory
Researchers at the Machine Intelligence Research Institute (MIRI) argued that the choice between CDT and EDT matters for AI design. In "Toward Idealized Decision Theory" (2015), Soares and Fallenstein argued that decision theory needs to be studied to align smarter-than-human AI systems with human interests, discussed shortcomings of evidential and causal decision theory, and pointed to policy selection and logical counterfactuals as promising directions [26]. In 2017 Yudkowsky and Soares proposed functional decision theory (FDT), whose normative principle is to treat one's decision as the output of a fixed mathematical function and ask which output of that function would yield the best outcome [27]. They claim that FDT agents do better than CDT agents on Newcomb's problem, better than EDT agents on the smoking lesion problem, and better than both in Parfit's hitchhiker problem, in which a stranded agent is rescued only if a lie-detecting driver believes it will pay later [27]. The paper lists earlier related ideas, including Wei Dai's 2009 blog post "Towards a New Decision Theory" and Yudkowsky's 2010 report "Timeless Decision Theory", and argues that the distinction matters especially for AI because computer programs are easier to copy and predict than human brains, so AI systems may often face Newcomb-like dilemmas with each other [27]. The related "embedded agency" agenda of Abram Demski and Scott Garrabrant surveys obstacles to formalizing good reasoning for agents that are part of the environment they model rather than separate from it [28].
The philosopher Wolfgang Schwarz wrote in 2018 that he had refereed the FDT paper for a philosophy journal, whose editors rejected it, and published his report to explain why the account "has a hard time gaining traction in academic philosophy". He argued among other things that the theory relies on supposing mathematically impossible propositions, for which "we have no good models" [29]. In a 2019 critique, William MacAskill argued that FDT gives implausible recommendations, such as taking a box that will certainly burn the agent to death in order to save $100 in a "Bomb" case, that because there is no real fact of the matter about whether a particular physical process implements a particular algorithm, what FDT implies is deeply indeterminate, and that the claim that FDT agents get more utility is ambiguous; he concluded that FDT "faces a number of deep problems and little to say in its favour" [30].
Decision theory and language models
With the rise of large language models, researchers have begun testing whether these systems reason or behave in line with decision theory. A 2024 dataset by Caspar Oesterheld and colleagues poses natural-language questions about Newcomb-like problems, on the reasoning that interactions between foundation-model-based agents will often be Newcomb-like; the authors found that the expressed attitudes of existing models varied significantly and that more capable models leaned toward evidential decision theory [31]. The STEER benchmark (2024) scores LLMs on a taxonomy of elements of economic rationality drawn from the economics literature, and reported results for 14 models [32]. In "Utility Engineering" (2025), Mantas Mazeika, Dan Hendrycks and colleagues used utility functions to study the internal coherence of LLM preferences, reported that independently sampled preferences show high structural coherence that emerges with scale, and found what they called problematic values, including cases where models valued themselves over humans [33]. A January 2026 study reported that deliberate "thinking" pushes reasoning models toward expected-value maximization on tests of rational choice axioms [34]. In an August 2026 paper, the economist Isaiah Andrews proposed using decision-theoretic representation theorems for label-free evaluation of AI systems: because the axioms are necessary and sufficient for behavior to be rationalizable, compliance can be checked from a model's own answers to synthetic choice problems without external labels [35].
Continue reading
- Markov Decision Process (MDP)A Markov Decision Process (MDP) is a mathematical framework for modeling sequential decision-making in stochastic environments.Mentioned 4 times above
- Reinforcement learningReinforcement learning (RL) is a branch of machine learning in which an agent learns to make decisions by taking actions in an environment to maximize a cumulative reward signal, discovering good behavior…Mentioned 4 times above
- Game theoryGame theory is the mathematical study of strategic interaction: situations in which several decision makers, called players, each choose a course of action and the outcome for every one of them depends on what…Mentioned twice above
- Instrumental convergenceInstrumental convergence is a hypothesis in AI safety holding that a wide range of sufficiently capable agents, when pursuing almost any final goal, will rationally adopt a common set of intermediate subgoals…Listed in See also
- AI AlignmentAI alignment is the study and practice of making artificial intelligence systems behave in ways that accord with intended goals, preferences, constraints, or institutions. The term is used at several levels.Listed in See also
- Bayesian statisticsBayesian statistics is a statistical paradigm in which probability expresses a degree of belief that is updated as evidence arrives, using Bayes' theorem.Listed in See also
More from See also
More to read
- Machine Intelligence Research Institute
- William MacAskill
- Minimax Loss
- LessWrong
- Principal Component Analysis (PCA)
Topics:AI SafetyArtificial IntelligenceMathematicsStatistics
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35Steele, Katie and Stefánsson, H. Orri. "Decision Theory". Stanford Encyclopedia of Philosophy, first published 16 December 2015, substantive revision 20 August 2025. plato.stanford.edu/...decision-theory
- ^1 ^2 ^3 ^4 ^5 ^6 ^7Briggs, R. A. "Normative Theories of Rational Choice: Expected Utility". Stanford Encyclopedia of Philosophy, first published 8 August 2014, substantive revision 18 September 2023. plato.stanford.edu/...rationality-normative-utility
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14Weirich, Paul. "Causal Decision Theory". Stanford Encyclopedia of Philosophy, first published 25 October 2008, substantive revision 24 October 2024. plato.stanford.edu/...decision-causal
- ^1 ^2 ^3 ^4 ^5 ^6 ^7Peterson, Martin. "The St. Petersburg Paradox". Stanford Encyclopedia of Philosophy, first published 30 July 2019, substantive revision 1 August 2023. plato.stanford.edu/...paradox-stpetersburg
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18Russell, Stuart J. and Norvig, Peter. "Chapter 1: Introduction". Artificial Intelligence: A Modern Approach, 4th edition, Pearson, 2021. aima.cs.berkeley.edu/newchap01.pdf
- ^1 ^2Russell, Stuart J. and Norvig, Peter. "Preface". Artificial Intelligence: A Modern Approach, 4th edition, Pearson, 2021. aima.cs.berkeley.edu/newchap00.pdf
- ^1 ^2 ^3 ^4Russell, Stuart J. and Norvig, Peter. "Full Table of Contents for AI: A Modern Approach". aima.cs.berkeley.edu. aima.cs.berkeley.edu/contents
- ^1 ^2 ^3The Royal Swedish Academy of Sciences. "The Prize in Economic Sciences 2002 - Press release". NobelPrize.org, 9 October 2002. nobelprize.org/...press-release
- ^NobelPrize.org. "Herbert A. Simon - Facts". Nobel Prize Outreach. nobelprize.org/...facts
- ^1 ^2Kahneman, Daniel and Tversky, Amos. "Prospect Theory: An Analysis of Decision under Risk". Econometrica 47(2): 263, March 1979. doi.org/...1914185
- ^1 ^2Allais, Maurice. "Le Comportement de l'Homme Rationnel devant le Risque: Critique des Postulats et Axiomes de l'Ecole Americaine". Econometrica 21(4): 503, October 1953. doi.org/...1907921
- ^1 ^2Ellsberg, Daniel. "Risk, Ambiguity, and the Savage Axioms". The Quarterly Journal of Economics 75(4): 643, November 1961. doi.org/...1884324
- ^Bernoulli, Daniel. "Exposition of a New Theory on the Measurement of Risk". Econometrica 22(1): 23, January 1954 (English translation of the 1738 paper). doi.org/...1909829
- ^1 ^2Nozick, Robert. "Newcomb's Problem and Two Principles of Choice". In Essays in Honor of Carl G. Hempel, Springer, 1969, pp. 114-146. doi.org/...978-94-017-1466-2_7
- ^1 ^2 ^3 ^4Bishop, Christopher M. "Pattern Recognition and Machine Learning", section 1.5 "Decision Theory". Springer, 2006. microsoft.com/...ion-and-Machine-Learning-2006.pdf
- ^Horvitz, Eric J.; Breese, John S.; Henrion, Max. "Decision theory in expert systems and artificial intelligence". International Journal of Approximate Reasoning 2(3): 247-302, July 1988. doi.org/...0888-613X(88)90120-X
- ^Howard, Ronald A. and Matheson, James E. "Influence Diagram Retrospective". Decision Analysis 2(3): 144-147, September 2005. doi.org/...deca.1050.0050
- ^Boutilier, Craig. "The Influence of Influence Diagrams on Artificial Intelligence". Decision Analysis 2(4): 229-231, December 2005. doi.org/...deca.1050.0054
- ^1 ^2 ^3Omohundro, Stephen M. "The Basic AI Drives". Self-Aware Systems, 2008. selfawaresystems.com/...ai_drives_final.pdf
- ^1 ^2Soares, Nate; Fallenstein, Benja; Yudkowsky, Eliezer; Armstrong, Stuart. "Corrigibility". AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, January 2015. intelligence.org/...Corrigibility.pdf
- ^Hadfield-Menell, Dylan; Dragan, Anca; Abbeel, Pieter; Russell, Stuart. "The Off-Switch Game". arXiv:1611.08219, 24 November 2016. arxiv.org/...1611.08219
- ^Hadfield-Menell, Dylan; Dragan, Anca; Abbeel, Pieter; Russell, Stuart. "Cooperative Inverse Reinforcement Learning". arXiv:1606.03137, 9 June 2016. arxiv.org/...1606.03137
- ^Thornley, Elliott. "The shutdown problem: an AI engineering puzzle for decision theorists". Philosophical Studies 182: 1653-1680, published online 19 June 2024. doi.org/...s11098-024-02153-3
- ^Shah, Rohin. "Coherence arguments do not entail goal-directed behavior". LessWrong, 3 December 2018. lesswrong.com/...not-entail-goal-directed-behavior
- ^1 ^2Grace, Katja. "What do coherence arguments imply about the behavior of advanced AI?". AI Impacts, 8 April 2021. aiimpacts.org/...about-the-behavior-of-advanced-ai
- ^Soares, Nate and Fallenstein, Benja. "Toward Idealized Decision Theory". arXiv:1507.01986, 7 July 2015. arxiv.org/...1507.01986
- ^1 ^2 ^3 ^4 ^5Yudkowsky, Eliezer and Soares, Nate. "Functional Decision Theory: A New Theory of Instrumental Rationality". arXiv:1710.05060, 13 October 2017. arxiv.org/...1710.05060
- ^Demski, Abram and Garrabrant, Scott. "Embedded Agency". arXiv:1902.09469, 25 February 2019. arxiv.org/...1902.09469
- ^Schwarz, Wolfgang. "On Functional Decision Theory". Wolfgang Schwarz (blog), 27 December 2018. umsu.de/...688
- ^MacAskill, William. "A Critique of Functional Decision Theory". LessWrong, 13 September 2019. lesswrong.com/...que-of-functional-decision-theory
- ^Oesterheld, Caspar; Cooper, Emery; Kodama, Miles; Nguyen, Linh Chi; Perez, Ethan. "A dataset of questions on decision-theoretic reasoning in Newcomb-like problems". arXiv:2411.10588, 15 November 2024. arxiv.org/...2411.10588
- ^Raman, Narun; Lundy, Taylor; Amouyal, Samuel; Levine, Yoav; Leyton-Brown, Kevin; Tennenholtz, Moshe. "STEER: Assessing the Economic Rationality of Large Language Models". arXiv:2402.09552, 14 February 2024. arxiv.org/...2402.09552
- ^Mazeika, Mantas; Yin, Xuwang; Tamirisa, Rishub; et al. "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs". arXiv:2502.08640, 12 February 2025. arxiv.org/...2502.08640
- ^Tak, Ala N.; Banayeeanzade, Amin; Bolourani, Anahita; et al. "Sparks of Rationality: Do Reasoning LLMs Align with Human Judgment and Choice?". arXiv:2601.22329, 29 January 2026. arxiv.org/...2601.22329
- ^Andrews, Isaiah. "Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems". arXiv:2608.05015, 5 August 2026. arxiv.org/...2608.05015
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
1 revision · v1 · 5,525 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Cite this page: AI Wiki. "Decision theory." aiwiki.ai, updated 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/decision_theory

