Citation and evidence

Value function

22 min full readUpdated 15 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

Machine LearningMathematicsReinforcement Learning

Cite this article

A value function in reinforcement learning (RL) is a function that assigns to each state the expected long-run return an agent will collect if it starts in that state and then behaves according to a given policy. Written $v_\pi(s)$ (or $V^\pi(s)$), the state-value function answers the question "how good is it to be here, given how I act?" Its close relative, the state-action value function $q_\pi(s, a)$ (the Q-function), also conditions on the first action taken. Sutton and Barto's textbook names the value function as one of four main subelements of a reinforcement learning system, alongside a policy, a reward signal and, optionally, a model of the environment. They call a method for efficiently estimating values the most important component of almost all the algorithms they consider, and write that "the central role of value estimation is arguably the most important thing that has been learned about reinforcement learning over the last six decades" [1].

The idea predates machine learning. In dynamic programming and optimal control, the same object is called the optimal return function or the cost-to-go function, and it is the quantity that the Bellman equation defines recursively [1][2]. Value functions are what TD-Gammon learned to play backgammon, what the "value network" of AlphaGo estimated to evaluate Go positions, and what the critic in actor-critic algorithms such as A3C and PPO predicts [3][4][5][6].

Definition

Value functions are defined for a Markov decision process (MDP), in which an agent observes a state $S_t$, picks an action $A_t$, receives a reward $R_{t+1}$ and moves to a new state $S_{t+1}$. The agent's aim is to maximize the expected return, the discounted sum of future rewards:

Gt=Rt+1+γRt+2+γ2Rt+3+⋯=∑k=0∞γkRt+k+1G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}

where $\gamma$, with $0 \le \gamma \le 1$, is the discount factor [1].

A policy $\pi(a \mid s)$ gives the probability of choosing action $a$ in state $s$. The value of state $s$ under policy $\pi$ is the expected return when the agent starts in $s$ and follows $\pi$ afterwards:

vπ(s)=Eπ[Gt∣St=s]=Eπ[∑k=0∞γkRt+k+1 | St=s]v_\pi(s) = \mathbb{E}_\pi\left[ G_t \mid S_t = s \right] = \mathbb{E}_\pi\left[ \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} \,\middle|\, S_t = s \right]

Sutton and Barto call $v_\pi$ the state-value function for policy $\pi$, and note that the value of a terminal state is always zero [1]. Because the expectation depends on how the agent will act, a value function is always defined with respect to a particular policy; the same state can be valuable under one policy and poor under another.

FunctionNotationMeaning
State-value function$v_\pi(s)$Expected return from state $s$ when following $\pi$
Action-value function (Q-function)$q_\pi(s, a)$Expected return from $s$ after taking action $a$, then following $\pi$
Optimal state-value function$v_*(s) = \max_\pi v_\pi(s)$Best expected return achievable from $s$ by any policy
Optimal action-value function$q_*(s, a) = \max_\pi q_\pi(s, a)$Best expected return after taking $a$ in $s$
Advantage function$A_\pi(s, a) = q_\pi(s, a) - v_\pi(s)$How much better action $a$ is than the policy's average behavior in $s$

Expanded article table

The two main functions are linked. The state value is the policy-weighted average of the action values, $v_\pi(s) = \sum_a \pi(a \mid s), q_\pi(s, a)$, and with a model of the environment's dynamics $p(s', r \mid s, a)$, an action value can be computed from state values one step ahead: $q_\pi(s, a) = \sum_{s', r} p(s', r \mid s, a)\left[r + \gamma v_\pi(s')\right]$ [1]. The advantage function is used in policy-gradient methods because it measures whether an action is better or worse than what the policy does by default [7].

Rewards versus values

Sutton and Barto draw a sharp distinction between rewards and values. A reward is the immediate signal the environment sends after each step; a value is a prediction of the total reward that will follow. A state can yield a low immediate reward and still have high value because it is reliably followed by rewarding states, and the reverse can also hold. In the book's framing, rewards are primary and values are secondary (there would be no values without rewards), yet decisions are made on the basis of values, because actions that lead to high-value states collect the most reward over the long run. Values are also much harder to obtain: rewards come directly from the environment, whereas values must be estimated and re-estimated from the agent's experience [1].

"Model Based Reinforcement Learning: Policy Iteration, Value Iteration, and Dynamic Programming", video from Steve Brunton, published January 7, 2022. Watch on YouTube.

Bellman equations

The value function satisfies a recursive consistency condition that links the value of a state to the values of its possible successors. For any policy $\pi$, the Bellman equation for $v_\pi$ is

vπ(s)=∑aπ(a∣s)∑s′,rp(s′,r∣s,a)[r+γ vπ(s′)]v_\pi(s) = \sum_a \pi(a \mid s) \sum_{s', r} p(s', r \mid s, a)\left[ r + \gamma\, v_\pi(s') \right]

It says that the value of a state equals the expected immediate reward plus the discounted value of the expected next state, averaged over the policy's action choices and the environment's responses. $v_\pi$ is the unique solution of this equation [1]. For a finite MDP the equation is a system of linear equations, one per state, which is why policy evaluation can in principle be done by solving a linear system [1].

The optimal value function satisfies the Bellman optimality equation, in which the average over the policy is replaced by a maximum over actions:

v∗(s)=max⁡a∑s′,rp(s′,r∣s,a)[r+γ v∗(s′)]v_*(s) = \max_a \sum_{s', r} p(s', r \mid s, a)\left[ r + \gamma\, v_*(s') \right]

Once $v_$ is known, an optimal policy follows from a one-step search: in each state, any action that attains the maximum on the right-hand side is optimal, so a policy that is greedy with respect to $v_$ is optimal. This works because $v_$ already accounts for the consequences of all future behavior. Choosing actions from $v_$ alone does require knowing the dynamics $p$, which is the main practical difference from $q_$: with the optimal action-value function the agent can pick $\arg\max_a q_(s, a)$ directly without any knowledge of successor states [1].

According to Sutton and Barto, the Bellman optimality equation was popularized by Richard Bellman in his 1957 book on dynamic programming, where he called it the "basic functional equation"; its counterpart for continuous time and state is the Hamilton-Jacobi-Bellman equation [1].

Computing value functions with a model

When the full MDP (transition probabilities and expected rewards) is known, dynamic programming computes value functions exactly or to any desired accuracy [1].

Policy evaluation

Iterative policy evaluation turns the Bellman equation into an update rule. Starting from an arbitrary estimate $v_0$ (with terminal states fixed at zero), each sweep over the states applies

vk+1(s)=∑aπ(a∣s)∑s′,rp(s′,r∣s,a)[r+γ vk(s′)]v_{k+1}(s) = \sum_a \pi(a \mid s) \sum_{s', r} p(s', r \mid s, a)\left[ r + \gamma\, v_k(s') \right]

and the sequence $v_k$ converges to $v_\pi$ as $k$ grows. Each such step is called an expected update, because it uses the expectation over all possible next states rather than a single sample [1].

Policy improvement and policy iteration

Computing $v_\pi$ is useful because it reveals how to improve $\pi$. The policy improvement theorem shows that the policy obtained by acting greedily with respect to $v_\pi$ (picking in each state the action with the highest one-step lookahead value) is at least as good as $\pi$ in every state, and strictly better unless $\pi$ is already optimal. Alternating full evaluation and greedy improvement gives policy iteration, which Sutton and Barto credit to Ronald Howard (1960). For a finite MDP it reaches an optimal policy and $v_*$ in a finite number of iterations [1].

Value iteration

Value iteration truncates evaluation to a single sweep and folds the improvement step into the update, turning the Bellman optimality equation into an update rule:

vk+1(s)=max⁡a∑s′,rp(s′,r∣s,a)[r+γ vk(s′)]v_{k+1}(s) = \max_a \sum_{s', r} p(s', r \mid s, a)\left[ r + \gamma\, v_k(s') \right]

For any starting point, the sequence converges to $v_$ under the same conditions that guarantee $v_$ exists. Sutton and Barto use the term generalized policy iteration for the general pattern in which an evaluation process and an improvement process interact, the value function being pushed toward the value of the current policy while the policy is pushed toward greediness with respect to the value function; they argue that almost all reinforcement learning methods fit this description [1].

Dynamic programming is limited by what Bellman called the curse of dimensionality: its computational requirements grow exponentially with the number of state variables [1]. That limit, together with the frequent lack of a known model, motivates learning value functions from sampled experience.

Learning value functions from experience

Model-free methods estimate $v_\pi$ from sequences of states, actions and rewards generated by actually following $\pi$, without knowing $p$ [1].

Monte Carlo methods

Monte Carlo prediction estimates the value of a state by averaging the returns observed after visits to that state. First-visit Monte Carlo averages the returns following the first visit to $s$ in each episode; every-visit Monte Carlo averages over all visits. Both converge to $v_\pi(s)$ as the number of visits grows. An incremental version updates the estimate toward each observed return:

V(St)←V(St)+α[Gt−V(St)]V(S_t) \leftarrow V(S_t) + \alpha\left[ G_t - V(S_t) \right]

where $\alpha$ is a step size. Because $G_t$ is known only at the end of an episode, Monte Carlo methods apply to episodic tasks and must wait until an episode finishes before updating [1].

Temporal-difference learning

Temporal-difference learning (TD) updates an estimate from the next reward and the current estimate of the next state's value, without waiting for the final outcome. Its ideas go back to Samuel's checkers program and other early work; Richard Sutton's 1988 paper separated TD learning from control, treated it as a general prediction method, introduced the TD($\lambda$) algorithm and proved some of its convergence properties [1][8]. The simplest version, TD(0), is

V(St)←V(St)+α[Rt+1+γV(St+1)−V(St)]V(S_t) \leftarrow V(S_t) + \alpha\left[ R_{t+1} + \gamma V(S_{t+1}) - V(S_t) \right]

The bracketed quantity $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$ is the TD error. Updating an estimate partly from other estimates is called bootstrapping: TD bootstraps like dynamic programming but learns from samples like Monte Carlo. For a fixed policy, tabular TD(0) has been proved to converge to $v_\pi$ (in the mean for a small constant step size, and with probability 1 when the step size decreases appropriately) [1].

Between these two extremes lie n-step TD methods, which bootstrap after $n$ real rewards, and TD($\lambda$), which can be understood as averaging all the $n$-step updates, each weighted in proportion to $\lambda^{n-1}$ and normalized so the weights sum to 1. With $\lambda = 1$ the algorithm, also known as TD(1), behaves like a Monte Carlo method, and unlike the basic Monte Carlo methods it can also be applied to discounted continuing tasks [1].

PropertyDynamic programmingMonte CarloTD(0)
Needs a model of the environmentYesNoNo
Uses sampled experienceNoYesYes
Bootstraps from other estimatesYesNoYes
Update timingFull sweeps over statesEnd of episodeAfter every step
Works in continuing (non-episodic) tasksYesNo (as usually defined)Yes

Expanded article table

Function approximation

Tables with one entry per state stop working when the state space is large or continuous. Approximate methods represent the value function as a parameterized function $\hat{v}(s, \mathbf{w}) \approx v_\pi(s)$ with a weight vector $\mathbf{w}$ much smaller than the number of states, so that an update to one state also changes the estimates for others [1]. The parameterization can be linear in features, $\hat{v}(s, \mathbf{w}) = \mathbf{w}^\top \mathbf{x}(s)$, or a nonlinear model such as a neural network.

A common objective is the mean square value error, weighted by how often each state is visited under the policy:

VE‾(w)=∑sμ(s)[vπ(s)−v^(s,w)]2\overline{\mathrm{VE}}(\mathbf{w}) = \sum_{s} \mu(s)\left[ v_\pi(s) - \hat{v}(s, \mathbf{w}) \right]^2

where $\mu$ is the on-policy distribution over states [1]. Gradient Monte Carlo minimizes this objective with stochastic gradient descent, using the observed return as the target. Semi-gradient TD(0) uses the bootstrapped target instead,

w←w+α[Rt+1+γv^(St+1,w)−v^(St,w)]∇v^(St,w)\mathbf{w} \leftarrow \mathbf{w} + \alpha\left[ R_{t+1} + \gamma \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_t, \mathbf{w}) \right] \nabla \hat{v}(S_t, \mathbf{w})

and is called "semi-gradient" because it ignores the dependence of the target on $\mathbf{w}$ [1]. In deep RL the same idea appears as a regression problem: the GAE paper, for example, fits a neural value function $V_\phi$ by minimizing the squared error to empirical discounted returns, which it notes is sometimes called the Monte Carlo or TD(1) approach [7].

Convergence and instability

John Tsitsiklis and Benjamin Van Roy proved in 1997 that TD($\lambda$) with a linear function approximator, updated online along a single trajectory of an irreducible aperiodic Markov chain, converges with probability one, and they characterized the limit and bounded its approximation error [2]. The same paper showed that divergence can occur when updates are not based on trajectories of the Markov chain, and gave an example of divergence with a nonlinear function approximator [2].

Sutton and Barto summarize the danger as the deadly triad: instability and divergence can arise whenever function approximation, bootstrapping and off-policy training are combined, and the problem occurs even in planning with a fully known environment. If only two of the three are present, instability can be avoided [1]. DQN, which learns an action-value function, is an example of how deep RL systems cope in practice: its experience replay removed one source of instability, and a duplicate target network, refreshed only every $C$ updates, supplied bootstrapping targets that did not shift with every weight update [1].

Value functions in policy-based methods

Methods that optimize a parameterized policy directly, such as policy gradient algorithms, do not need a value function to choose actions, but many of them learn one anyway to reduce the variance of their gradient estimates [6].

Baselines and critics

In the REINFORCE family of algorithms, subtracting a baseline from the return leaves the gradient estimate unbiased while lowering its variance, and a learned estimate of the state value is a natural choice of baseline [1][5]. The A3C paper (Mnih and colleagues, 2016) describes this as a critic: when an approximate value function $V^\pi(s_t)$ serves as the baseline, the scaled quantity is an estimate of the advantage, and the method can be viewed as an actor-critic architecture in which the policy is the actor and the baseline is the critic [5]. In A3C the policy and value function share network layers, with a softmax output for the policy and one linear output for $V(s_t)$ [5].

Generalized advantage estimation

John Schulman and colleagues' generalized advantage estimation (GAE), published at ICLR 2016, builds advantage estimates from the TD residuals of an approximate value function, $\delta_t^V = r_t + \gamma V(s_{t+1}) - V(s_t)$:

A^tGAE(γ,λ)=∑l=0∞(γλ)l δt+lV\hat{A}_t^{\mathrm{GAE}(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma \lambda)^l\, \delta_{t+l}^V

Setting $\lambda = 0$ gives the one-step TD residual, which has low variance but is biased unless $V$ is exact; $\lambda = 1$ gives the empirical return minus the value baseline, which has high variance but does not depend on the accuracy of $V$. The authors found empirically that the best $\lambda$ was much lower than the best $\gamma$, which they attribute to $\lambda$ introducing far less bias than $\gamma$ when the value function is reasonably accurate [7].

PPO

Proximal Policy Optimization (Schulman and colleagues, 2017) uses a learned state-value function to compute variance-reduced advantage estimates such as GAE. When the policy and value function share parameters, PPO maximizes a combined objective consisting of the clipped policy surrogate, minus a squared-error value loss $(V_\theta(s_t) - V_t^{\mathrm{targ}})^2$ weighted by a coefficient, plus an entropy bonus [6].

Value functions in game playing

Game programs used approximate value functions, usually called evaluation functions, decades before deep reinforcement learning.

Shannon and Samuel

In describing how a computer could play chess, Claude Shannon (1950) suggested an evaluation function that weighed the long-term advantages and disadvantages of positions. Arthur Samuel's checkers programs (1959, 1967) scored the end positions of a lookahead search with a "scoring polynomial," a linear combination of features, and adjusted it from experience. Sutton and Barto describe Samuel's player as the earliest example they know of in which function approximation was used to learn value functions, and as an early use of what would now be called temporal-difference learning [1].

TD-Gammon

Gerald Tesauro's TD-Gammon was a neural network that, in Tesauro's words, "trains itself to be an evaluation function for the game of backgammon by playing against itself and learning from the outcome" [3]. The network output an estimate of the expected outcome of each position and was trained with TD($\lambda$); during self-play training it scored every legal move and chose the one with the maximum expected outcome for the side to move [3]. In a 40-game session, version 2.1 came within one point of former world champion Bill Robertie, who assessed it as playing at a strong master level extremely close to the world's best human players [3]. Sutton and Barto summarize the program's results across versions [1]:

VersionHidden unitsTraining gamesOpponentsResult
TD-Gammon 0.040300,000Other programsTied for best
TD-Gammon 1.080300,000Robertie, Magriel and others-13 points in 51 games
TD-Gammon 2.040800,000Various grandmasters-7 points in 38 games
TD-Gammon 2.1801,500,000Robertie-1 point in 40 games
TD-Gammon 3.0801,500,000Kazaros+6 points in 20 games

Expanded article table

AlphaGo

The 2016 AlphaGo paper framed Go as a problem of estimating the optimal value function $v_*(s)$, which determines the outcome of every position under perfect play. Because exhaustive search is infeasible, AlphaGo truncated its search with a "value network" $v_\theta(s)$ trained by regression on state-outcome pairs to approximate the value function of its strongest policy [4]. Training on positions from complete human games caused the network to memorize outcomes, so DeepMind generated a self-play data set of 30 million distinct positions, each sampled from a separate game. The resulting value network approached the accuracy of Monte Carlo rollouts with its reinforcement-learned policy while using 15,000 times less computation [4].

Extensions

Universal value function approximators (UVFAs), introduced by Tom Schaul, Daniel Horgan, Karol Gregor and David Silver at ICML 2015, generalize the value function to depend on a goal as well as a state, $V(s, g)$. The authors learned them by factoring observed values into separate state and goal embeddings, and showed that UVFAs can generalize to goals not seen during training [9].

Distributional reinforcement learning replaces the expected return with its full probability distribution. Marc Bellemare, Will Dabney and Rémi Munos argued at ICML 2017 for the importance of this "value distribution", the distribution of the random return an agent receives, in contrast to the usual approach of modeling only its expectation (the value). They applied Bellman's equation to learning approximate value distributions and reported state-of-the-art results on the Arcade Learning Environment [15].

In operations research and control, where problems are usually phrased as minimizing cost rather than maximizing reward, the value function is called the cost-to-go function. Tsitsiklis and Van Roy describe predictions of long-term future cost as a function of state as the cost-to-go function central to dynamic programming and optimal control, and give the net present value of a company (a discounted sum of its future cash flows) as an analogous prediction problem [2]. Sutton and Barto credit Daniela de Farias (with Van Roy) with foundational work on the linear programming approach to reinforcement learning [1].

Neuroscience

Value learning has also been used as a model of reward learning in the brain. Under the reward prediction error hypothesis of dopamine neuron activity, one function of the phasic activity of dopamine-producing neurons in mammals is to broadcast an error between an old and a new estimate of expected future reward, that is, something like a TD error. Sutton and Barto write that the hypothesis was first explicitly stated by Read Montague, Peter Dayan and Terrence Sejnowski in 1996, drawing on experiments carried out in Wolfram Schultz's laboratory in the 1980s and early 1990s, and they call the dopamine and TD-error parallel the most remarkable point of contact between reinforcement learning and neuroscience [1]. A 1997 review in Science by Schultz, Dayan and Montague summarized evidence that dopamine neurons in primates signal changes or errors in the predictions of future rewarding events, and argued that these findings can be understood through quantitative theories of adaptive optimizing control [10].

Large language models

PPO-based reinforcement learning from human feedback brought learned value functions into large language model training, since PPO trains a value model alongside the policy. In OpenAI's InstructGPT work (Ouyang and colleagues, 2022), the PPO value function was initialized from the reward model; for every policy size the team used a 6B reward model and a 6B value function, and for both the unembedding layer of the underlying GPT-3 architecture was replaced with a projection layer that outputs a scalar [11].

Training a separate value model for language-model RL is costly. The DeepSeekMath paper (Shao and colleagues, 2024) noted that the value function in PPO is typically another model of comparable size to the policy, bringing a substantial memory and computational burden, and that training it to be accurate at every token is complicated when, as is common, only the last token receives a reward score. Its Group Relative Policy Optimization (GRPO) drops the value model and uses the average reward of several sampled outputs for the same question as the baseline [12].

Work from ByteDance Seed in 2025 argued that a learned value function can still work for long chain-of-thought reasoning if the problems in training it are addressed. One paper identified value initialization bias and reward signal decay as root causes of PPO's failure on long-chain-of-thought tasks and proposed Value-Calibrated PPO, which pretrains the value model and decouples the GAE computation between actor and critic [13]. A follow-up framework from an overlapping team, VAPO, was built on a Qwen 32B pretrained model and addressed value model bias, heterogeneous sequence lengths and sparse rewards. It reported a score of 60.4 on AIME 2024, more than 10 points above the previously reported results of DeepSeek-R1-Zero-Qwen-32B and DAPO under the authors' identical settings [14].

History

YearDevelopmentSource
1950Shannon proposes evaluation functions for computer chess[1]
1957Bellman's Dynamic Programming popularizes the "basic functional equation" for optimal values[1]
1959Samuel publishes his checkers player, which learns a linear value function[1]
1960Howard devises policy iteration for MDPs[1]
1988Sutton's paper treats TD learning as a general prediction method and introduces TD($\lambda$)[1][8]
1995Tesauro describes TD-Gammon in Communications of the ACM[3]
1996-1997Dopamine activity linked to TD errors (Montague, Dayan and Sejnowski; Schultz, Dayan and Montague)[1][10]
1997Tsitsiklis and Van Roy prove convergence of linear TD($\lambda$)[2]
2015Universal value function approximators[9]
2016AlphaGo's value network; A3C; GAE[4][5][7]
2017PPO combines policy and value losses; distributional RL learns the value distribution[6][15]
2022InstructGPT trains a 6B value function for RLHF[11]
2024GRPO removes the value model from PPO-style training[12]
2025VC-PPO and VAPO return to value-based RL for long reasoning[13][14]

Expanded article table

Continue reading

More from See also

More to read

Topics:Machine LearningMathematicsReinforcement Learning

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36Sutton, Richard S.; Barto, Andrew G. "Reinforcement Learning: An Introduction" (2nd edition). MIT Press, 2018. incompleteideas.net/...the-book-2nd
  2. ^1 ^2 ^3 ^4 ^5Tsitsiklis, John N.; Van Roy, Benjamin. "An Analysis of Temporal-Difference Learning with Function Approximation". IEEE Transactions on Automatic Control, vol. 42, no. 5, pp. 674-690, May 1997. web.mit.edu/...J063-97-bvr-td.pdf
  3. ^1 ^2 ^3 ^4 ^5Tesauro, Gerald. "Temporal Difference Learning and TD-Gammon". Communications of the ACM, vol. 38, no. 3, pp. 58-68, March 1995. doi.org/...203330.203343
  4. ^1 ^2 ^3 ^4Silver, David; Huang, Aja; Maddison, Chris J.; et al. "Mastering the game of Go with deep neural networks and tree search". Nature, vol. 529, pp. 484-489, 2016. doi.org/...nature16961
  5. ^1 ^2 ^3 ^4 ^5Mnih, Volodymyr; Badia, Adria Puigdomenech; Mirza, Mehdi; et al. "Asynchronous Methods for Deep Reinforcement Learning". Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016. arxiv.org/...1602.01783
  6. ^1 ^2 ^3 ^4Schulman, John; Wolski, Filip; Dhariwal, Prafulla; Radford, Alec; Klimov, Oleg. "Proximal Policy Optimization Algorithms". arXiv, July 2017. arxiv.org/...1707.06347
  7. ^1 ^2 ^3 ^4Schulman, John; Moritz, Philipp; Levine, Sergey; Jordan, Michael; Abbeel, Pieter. "High-Dimensional Continuous Control Using Generalized Advantage Estimation". ICLR 2016 (arXiv, June 2015). arxiv.org/...1506.02438
  8. ^1 ^2Sutton, Richard S. "Learning to Predict by the Methods of Temporal Differences". Machine Learning, vol. 3, no. 1, pp. 9-44, August 1988. doi.org/...BF00115009
  9. ^1 ^2Schaul, Tom; Horgan, Daniel; Gregor, Karol; Silver, David. "Universal Value Function Approximators". Proceedings of the 32nd International Conference on Machine Learning, PMLR 37, pp. 1312-1320, 2015. proceedings.mlr.press/...schaul15
  10. ^1 ^2Schultz, Wolfram; Dayan, Peter; Montague, P. Read. "A Neural Substrate of Prediction and Reward". Science, vol. 275, no. 5306, pp. 1593-1599, 14 March 1997. pubmed.ncbi.nlm.nih.gov/9054347
  11. ^1 ^2Ouyang, Long; Wu, Jeff; Jiang, Xu; et al. "Training language models to follow instructions with human feedback". arXiv, March 2022. arxiv.org/...2203.02155
  12. ^1 ^2Shao, Zhihong; Wang, Peiyi; Zhu, Qihao; et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models". arXiv, February 2024. arxiv.org/...2402.03300
  13. ^1 ^2Yuan, Yufeng; Yue, Yu; Zhu, Ruofei; Fan, Tiantian; Yan, Lin. "What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret". arXiv, March 2025. arxiv.org/...2503.01491
  14. ^1 ^2Yue, Yu; Yuan, Yufeng; Yu, Qiying; et al. "VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks". arXiv, April 2025. arxiv.org/...2504.05118
  15. ^1 ^2Bellemare, Marc G.; Dabney, Will; Munos, Rémi. "A Distributional Perspective on Reinforcement Learning". Proceedings of the 34th International Conference on Machine Learning, PMLR 70, pp. 449-458, 2017. arxiv.org/...1707.06887

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

2 revisions · v2 · 4,410 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 78 claims checked against 17 cited and independent sources; 0 material and 6 minor corrections applied before publication.

Cite this page: AI Wiki. "Value function." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/value_function

Edit article