Citation and evidence

Trust Region Policy Optimization

16 min full readUpdated 14 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

AlgorithmsReinforcement LearningTraining & Optimization

Cite this article

Trust Region Policy Optimization (TRPO) is a policy gradient algorithm for reinforcement learning that improves a stochastic policy by maximizing a surrogate objective while keeping the average KL divergence between the old and new policies below a fixed bound, the "trust region". It was introduced by John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan and Pieter Abbeel of the University of California, Berkeley in a paper posted to arXiv on 19 February 2015 and presented at the 2015 International Conference on Machine Learning (ICML).[1][2] The authors derived the method from a theoretical procedure with guaranteed monotonic improvement, then made a series of approximations to obtain a practical algorithm that is related to natural policy gradient methods and can train large nonlinear policies such as neural networks.[1]

The original paper used the same algorithm to learn simulated robot swimming, hopping and walking gaits and to play Atari games from screen images.[1] Later benchmark studies included TRPO among the algorithms they evaluated, and a 2020 ICLR study described it and PPO as two of the most popular deep policy gradient methods.[7][8] Its successor, Proximal Policy Optimization (PPO), was written by Schulman and colleagues at OpenAI in 2017 to keep TRPO's data efficiency and reliability while using only first-order optimization.[6] In 2025 ICML gave the TRPO paper a Test of Time Honorable Mention.[3]

Background

Step sizes in policy gradient methods

A policy gradient method adjusts the parameters $\theta$ of a stochastic policy $\pi_\theta(a \mid s)$ in the direction that increases expected return. The TRPO paper groups policy optimization methods into three families: policy iteration methods, which alternate between estimating the value function of the current policy and improving the policy; policy gradient methods, which estimate the gradient of expected return from sampled trajectories; and derivative-free methods such as the cross-entropy method (CEM) and covariance matrix adaptation (CMA), which treat return as a black-box function of the parameters.[1] The authors noted that derivative-free methods were often preferred in practice because they were simple and worked well, even though gradient-based optimization has much better sample complexity guarantees.[1]

The difficulty with ordinary gradient steps is choosing how far to move. OpenAI's Spinning Up documentation explains that vanilla policy gradient keeps new and old policies close in parameter space, but small differences in parameters can produce large differences in performance, so a single bad step can collapse the policy; this makes large step sizes dangerous and hurts sample efficiency.[11] TRPO addresses this by measuring the size of an update in terms of the policies' action distributions rather than their parameters.[11]

Natural policy gradient and conservative policy iteration

TRPO builds on two earlier lines of work. Sham Kakade's natural policy gradient (NeurIPS 2001) multiplies the policy gradient by the inverse of the Fisher information matrix, the Hessian of the KL divergence between policies, and treats the step size as a tuning parameter.[1][4] Kakade and Langford's conservative policy iteration (2002) gave an explicit lower bound on the improvement of a new policy, but only for policies formed as a mixture of the old policy and a greedy update; the TRPO authors described this policy class as unwieldy and restrictive in practice.[1] TRPO's main theoretical result extends that bound to general stochastic policies.[1]

"L4 TRPO and PPO (Foundations of Deep RL Series)", video from Pieter Abbeel, published August 25, 2021. Watch on YouTube.

Theory

Surrogate objective

Let $\eta(\pi)$ be the expected discounted return of policy $\pi$ in a Markov decision process with discount factor $\gamma \in (0, 1)$, and let $A_\pi(s, a) = Q_\pi(s, a) - V_\pi(s)$ be the advantage function, where $Q_\pi$ is the state-action value function and $V_\pi$ the state value function. Following Kakade and Langford, the paper uses the identity

η(π~)=η(π)+Es0,a0,…∼π~[∑t=0∞γtAπ(st,at)],\eta(\tilde{\pi}) = \eta(\pi) + \mathbb{E}_{s_0, a_0, \ldots \sim \tilde{\pi}}\left[\sum_{t=0}^{\infty} \gamma^t A_\pi(s_t, a_t)\right],

which expresses the return of a new policy $\tilde{\pi}$ in terms of the advantages of the old policy $\pi$.[1] Because the state distribution on the right depends on $\tilde{\pi}$, the expression is hard to optimize directly, so the paper introduces a local approximation that uses the old policy's discounted state visitation frequencies $\rho_\pi(s)$ instead:

Lπ(π~)=η(π)+∑sρπ(s)∑aπ~(a∣s)Aπ(s,a).L_\pi(\tilde{\pi}) = \eta(\pi) + \sum_s \rho_\pi(s) \sum_a \tilde{\pi}(a \mid s) A_\pi(s, a).

For a parameterized policy, $L$ matches $\eta$ to first order at the current parameters, so a small enough step that improves $L$ also improves $\eta$, but this alone does not say how large the step may be.[1]

Monotonic improvement bound

The paper's Theorem 1 bounds the true return of a new policy by the surrogate minus a penalty on how far the policy moved. With $\alpha = D_{\mathrm{TV}}^{\max}(\pi_{\mathrm{old}}, \pi_{\mathrm{new}})$, the largest total variation divergence between the two policies' action distributions over all states, and $\epsilon = \max_{s,a} |A_\pi(s, a)|$:

η(πnew)≥Lπold(πnew)−4ϵγ(1−γ)2α2.\eta(\pi_{\mathrm{new}}) \geq L_{\pi_{\mathrm{old}}}(\pi_{\mathrm{new}}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2.

Using the inequality $D_{\mathrm{TV}}(p | q)^2 \leq D_{\mathrm{KL}}(p | q)$, this gives $\eta(\tilde{\pi}) \geq L_\pi(\tilde{\pi}) - C , D_{\mathrm{KL}}^{\max}(\pi, \tilde{\pi})$ with $C = 4\epsilon\gamma / (1-\gamma)^2$.[1] Assuming exact advantage values, an algorithm that repeatedly maximizes the right-hand side (the paper's Algorithm 1) is guaranteed to produce a sequence of policies whose returns never decrease. The authors identify it as a minorization-maximization algorithm, the class of methods that also includes expectation maximization.[1]

From theory to a practical algorithm

Three approximations turn Algorithm 1 into TRPO:[1]

Theoretical procedurePractical TRPOReason given in the paper
Penalty $C , D_{\mathrm{KL}}^{\max}$ added to the objectiveHard constraint: KL at most $\delta$The theoretical $C$ gives prohibitively small steps, and a penalty coefficient is hard to choose robustly
Maximum KL over all statesAverage KL over states visited by the old policyThe maximum is impractical to estimate and optimize, with one constraint per state
Exact advantage valuesMonte Carlo estimates from sampled trajectoriesThe analysis omits estimation error for simplicity

Expanded article table

The resulting optimization problem, written with expectations, is

max⁡θ  Es∼ρθold, a∼q[πθ(a∣s)q(a∣s)Qθold(s,a)]subject toEs∼ρθold[DKL(πθold(⋅∣s) ∥ πθ(⋅∣s))]≤δ,\max_\theta \; \mathbb{E}_{s \sim \rho_{\theta_{\mathrm{old}}},\, a \sim q}\left[\frac{\pi_\theta(a \mid s)}{q(a \mid s)} Q_{\theta_{\mathrm{old}}}(s, a)\right] \quad \text{subject to} \quad \mathbb{E}_{s \sim \rho_{\theta_{\mathrm{old}}}}\left[D_{\mathrm{KL}}\big(\pi_{\theta_{\mathrm{old}}}(\cdot \mid s) \,\|\, \pi_\theta(\cdot \mid s)\big)\right] \leq \delta,

where $q$ is the distribution used to sample actions and the probability ratio is an importance sampling correction.[1] On the cart-pole task, where it was tractable, the authors also ran the maximum-KL version; it learned somewhat more slowly, which they took as evidence that the average-KL constraint has a similar effect to the theoretically justified one.[1]

Later work tightened the link between the average-KL constraint and the theory. In the 2017 Constrained Policy Optimization paper, Joshua Achiam, David Held, Aviv Tamar and Pieter Abbeel proved a bound relating the returns of two policies to an average divergence between them, and showed that the average-KL trust region update, previously motivated heuristically, has a worst-case performance degradation guarantee that depends on $\delta$.[10]

Algorithm

Sampling schemes

The paper describes two ways to collect data and estimate Q-values:[1]

SchemeHow it estimates Q-valuesRequirements
Single pathRuns the current policy and uses the discounted sum of future rewards along each trajectoryNo state resets; can run on a physical system
VinePicks a "rollout set" of states from trunk trajectories, tries several actions from each, and runs a short rollout after each action using common random numbersThe simulator must be reset to arbitrary states

Expanded article table

The vine method gives lower-variance advantage estimates for the same number of Q-value samples, at the cost of many more simulator calls.[1]

The update step

Each TRPO iteration collects state-action pairs with Monte Carlo Q-value estimates, builds the sample-average objective and constraint, and approximately solves the constrained problem.[1] The solver has two stages, described in the paper's Appendix C:[1]

  1. Search direction. Take a linear approximation of the objective, with gradient $g$, and a quadratic approximation of the KL constraint, $D_{\mathrm{KL}} \approx \frac{1}{2}(\theta - \theta_{\mathrm{old}})^\top A (\theta - \theta_{\mathrm{old}})$, where $A$ is the Fisher information matrix. The direction $s \approx A^{-1} g$ is found with the conjugate gradient algorithm, which needs only matrix-vector products $y \mapsto Ay$ and never forms $A$ or its inverse.
  2. Step length and line search. The largest step that satisfies the quadratic constraint is $\beta = \sqrt{2\delta / s^\top A s}$. Starting from $\theta_{\mathrm{old}} + \beta s$, the step is shrunk exponentially until the surrogate objective improves and the actual KL constraint holds. The authors wrote that without this line search the algorithm occasionally takes large steps that cause catastrophic drops in performance.

The paper computes the Fisher information matrix analytically as the Hessian of the KL divergence rather than as the covariance of sampled gradients (the "empirical FIM"). A Fisher-vector product costs about as much as a gradient computation. The authors found 10 conjugate gradient iterations effective, and by computing the products on about 10% of the data they made the natural gradient step add little cost beyond computing the gradient.[1]

Spinning Up summarizes the same update: without the line search the step would be exactly the natural policy gradient, and TRPO adds a backtracking line search because the Taylor approximations may produce a step that violates the KL constraint or fails to improve the surrogate.[11]

Relation to other update rules

The paper presents TRPO's constrained problem as a framework that contains several earlier methods as special cases:[1]

MethodHow it arises from the TRPO problem
Natural policy gradientLinear approximation to the objective, quadratic approximation to the KL constraint, and a fixed step size instead of enforcing the constraint at each update
Standard policy gradientLinear objective with a Euclidean ($\ell_2$) constraint on the parameter change
Policy iterationUnconstrained maximization of the surrogate $L$
Relative entropy policy search (REPS)Constrains state-action marginals, while TRPO constrains the conditional action distributions; REPS needs a costly nonlinear optimization in the inner loop

Expanded article table

The authors reported that enforcing the constraint at every update, the main difference from the natural policy gradient, significantly improved performance on larger problems.[1]

Experiments in the original paper

Simulated locomotion

The robotic locomotion experiments used the MuJoCo simulator with three planar robots: a swimmer (10-dimensional state), a hopper (12-dimensional) and a walker (18-dimensional), plus the classic cart-pole task. All experiments used $\delta = 0.01$.[1] TRPO was compared with CEM, CMA, the natural policy gradient (with its step size tuned by a sweep), and the empirical-FIM and max-KL variants.[1] Both single-path and vine TRPO solved all the problems and gave the best solutions. The natural gradient did well on the two easier problems but could not produce hopping or walking gaits that made forward progress, and the derivative-free methods did poorly on the larger problems.[1] The authors stressed that TRPO learned the gaits with general-purpose neural network policies and simple rewards, where most earlier locomotion work relied on hand-designed policy classes that encoded balance and stepping.[1]

Atari games

To test a partially observed task with high-dimensional inputs, the authors trained convolutional neural network policies (two convolutional layers and one fully connected layer, 33,500 parameters) on seven Atari games from screen images. The 500 iterations took about 30 hours, with slight variation between games, on a 16-core computer. Each variant was run once per game; the authors noted that performance varies substantially from run to run but said time constraints kept them from obtaining error statistics.[1]

GameDeep Q-learning (Mnih et al., 2013)TRPO single pathTRPO vine
Beam Rider40921425.2859.5
Breakout168.010.834.2
Enduro470534.6430.8
Pong20.020.920.9
Q*bert19521973.57732.5
Seaquest17051908.6788.4
Space Invaders581568.4450.2

Expanded article table

Source: Table 1 of the TRPO paper.[1] The authors acknowledged that TRPO beat the prior methods (DQN, and UCC-I, which combines Monte Carlo tree search with supervised training) on only some games, but argued that it consistently reached reasonable scores without being designed for the task, and that using one method for both locomotion and image-based games showed its generality.[1]

Later development

Generalized advantage estimation

In June 2015 the same group posted "High-Dimensional Continuous Control Using Generalized Advantage Estimation" (ICLR 2016). It paired TRPO policy updates with an exponentially weighted advantage estimator analogous to TD($\lambda$), and also used a trust region method to fit a neural network value function.[5] With this combination the authors learned running gaits for simulated 3D bipeds and quadrupeds and a stand-up behavior for a biped, using simulated experience equivalent to 1-2 weeks of real time for the biped tasks.[5] Schulman's 2016 doctoral thesis at Berkeley, advised by Abbeel, presents TRPO in its third chapter and generalized advantage estimation in its fourth.[14]

Benchmarks

The 2016 rllab benchmark paper by Yan Duan, Xi Chen, Rein Houthooft, John Schulman and Pieter Abbeel evaluated a range of algorithms on continuous control tasks. It found that truncated natural policy gradient (TNPG) and TRPO outperformed the other batch algorithms by a large margin on most tasks.[7] Compared with TNPG, TRPO's line search gave better control over each update and allowed larger KL step sizes; on the Swimmer task the best step size found by grid search was $\delta_{\mathrm{KL}} = 0.05$ for TNPG and $0.1$ for TRPO.[7]

Proximal Policy Optimization

The PPO paper (July 2017) described TRPO as "relatively complicated" and incompatible with architectures that include noise, such as dropout, or parameter sharing between the policy and value function or with auxiliary tasks.[6] It also observed that TRPO's theory points to a KL penalty rather than a constraint, but that a single penalty coefficient is hard to choose across problems, or even within one problem as it changes during learning.[6] PPO instead optimizes a clipped probability-ratio objective with several epochs of minibatch stochastic gradient ascent on each batch of data, and in the paper's continuous control comparison it outperformed tuned implementations of TRPO and other methods on almost all environments.[6]

TRPO (2015)PPO (2017)
How the update is limitedHard constraint on average KL divergenceClipped probability ratio in the objective (or an adaptive KL penalty)
OptimizerConjugate gradient plus line searchFirst-order stochastic gradient ascent
Passes over each batchOne constrained updateSeveral epochs of minibatch updates
Curvature informationFisher-vector productsNone

Expanded article table

Sources: the TRPO and PPO papers.[1][6]

A 2020 study by Logan Engstrom, Andrew Ilyas and colleagues at MIT and Two Sigma, presented at ICLR 2020, examined where PPO's advantage comes from. It found that "code-level optimizations", details found in implementations rather than in the core algorithm, were responsible for most of PPO's gain in cumulative reward over TRPO.[8] The study measured that TRPO enforced its mean-KL trust region precisely, as designed, while PPO's clipping did not by itself keep the probability ratio inside its bounds.[8] The authors also added the same optimizations to TRPO, a variant they called TRPO+. Once the use of the optimizations was held fixed, the choice of core algorithm did not appear to have a significant effect on reward:[8]

AlgorithmWalker2d-v2Hopper-v2Humanoid-v2
TRPO27912043586
TRPO+ (with PPO's code-level optimizations)305024661030
PPO-M (PPO without them)27352142674
PPO32922513806

Expanded article table

Mean rewards from Table 2 of Engstrom et al.; the paper reports 95% confidence intervals.[8] The authors concluded that changing the code-level optimizations affected performance more than choosing the PPO or TRPO update step.[8]

Other trust region methods

Several later algorithms kept TRPO's trust region idea while changing how it is computed or what it constrains:

  • ACKTR (2017), by Yuhuai Wu, Elman Mansimov, Shun Liao, Roger Grosse and Jimmy Ba, applies trust region optimization to both the actor and the critic using a Kronecker-factored approximation to the curvature (K-FAC). The authors described it as the first scalable trust region natural gradient method for actor-critic methods.[9]
  • Constrained Policy Optimization (CPO, ICML 2017) extends the trust region approach to constrained Markov decision processes, where a policy must also satisfy safety constraints, and inherits the trust region performance bound described above.[10]

Implementations

TRPO has reference implementations in several widely used libraries:

LibraryNotes
OpenAI Baselinestrpo_mpi, parallelized with MPI; examples cover Atari Pong and MuJoCo Ant[12]
Spinning UpTensorFlow implementation trpo_tf1; supports discrete and continuous action spaces[11]
Stable-Baselines3 ContribTRPO class for discrete and continuous action spaces; recurrent policies not supported[13]

Expanded article table

Spinning Up's defaults illustrate typical settings: a KL limit of 0.01, 10 conjugate gradient iterations, a damping coefficient of 0.1, up to 10 backtracking steps with a coefficient of 0.8, and a GAE $\lambda$ of 0.97.[11] It classifies TRPO as an on-policy algorithm that explores by sampling from its current stochastic policy, which tends to become less random during training and can get stuck in local optima.[11]

Recognition

At ICML 2025 the TRPO paper received a Test of Time Honorable Mention, alongside "Variational Inference with Normalizing Flows"; the Test of Time Award itself went to the 2015 batch normalization paper by Sergey Ioffe and Christian Szegedy.[3] The award listing names the authors in the order of the ICML proceedings version: John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan and Philipp Moritz.[2][3]

Continue reading

More to read

Topics:AlgorithmsReinforcement LearningTraining & Optimization

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel. "Trust Region Policy Optimization". arXiv:1502.05477, 19 February 2015 (revised 20 April 2017). arxiv.org/...1502.05477
  2. ^1 ^2John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz. "Trust Region Policy Optimization". Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:1889-1897, 2015. proceedings.mlr.press/...schulman15
  3. ^1 ^2 ^3ICML. "ICML 2025 Awards". icml.cc, 2025. icml.cc/...awards_detail
  4. ^Sham M. Kakade. "A Natural Policy Gradient". Advances in Neural Information Processing Systems 14, 2001. papers.nips.cc/...d358ecf194c56c69108433e-Abstract
  5. ^1 ^2John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, Pieter Abbeel. "High-Dimensional Continuous Control Using Generalized Advantage Estimation". ICLR 2016; arXiv:1506.02438, 8 June 2015. arxiv.org/...1506.02438
  6. ^1 ^2 ^3 ^4 ^5John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. "Proximal Policy Optimization Algorithms". arXiv:1707.06347, 20 July 2017. arxiv.org/...1707.06347
  7. ^1 ^2 ^3Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel. "Benchmarking Deep Reinforcement Learning for Continuous Control". ICML 2016; arXiv:1604.06778, 22 April 2016. arxiv.org/...1604.06778
  8. ^1 ^2 ^3 ^4 ^5 ^6Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry. "Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO". ICLR 2020; arXiv:2005.12729, 25 May 2020. arxiv.org/...2005.12729
  9. ^Yuhuai Wu, Elman Mansimov, Shun Liao, Roger Grosse, Jimmy Ba. "Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation". arXiv:1708.05144, 17 August 2017. arxiv.org/...1708.05144
  10. ^1 ^2Joshua Achiam, David Held, Aviv Tamar, Pieter Abbeel. "Constrained Policy Optimization". ICML 2017; arXiv:1705.10528, 30 May 2017. arxiv.org/...1705.10528
  11. ^1 ^2 ^3 ^4 ^5 ^6OpenAI. "Trust Region Policy Optimization". Spinning Up documentation. spinningup.openai.com/...trpo
  12. ^OpenAI. "trpo_mpi". OpenAI Baselines, GitHub. github.com/...trpo_mpi
  13. ^Stable-Baselines3 Contrib. "TRPO". Stable Baselines3 Contrib documentation. sb3-contrib.readthedocs.io/...trpo
  14. ^John Schulman. "Optimizing Expectations: From Deep Reinforcement Learning to Stochastic Computation Graphs". Technical Report UCB/EECS-2016-217, EECS Department, University of California, Berkeley, 16 December 2016. www2.eecs.berkeley.edu/...EECS-2016-217

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

2 revisions · v2 · 3,263 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 78 claims checked against 16 cited and independent sources; 0 material and 2 minor corrections applied before publication.

Cite this page: AI Wiki. "Trust Region Policy Optimization." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/trust_region_policy_optimization

Edit article