Stable Baselines
Stable Baselines is a family of open-source Python libraries that provide implementations of model-free deep reinforcement learning (RL) algorithms behind a common, scikit-learn-style interface. The original Stable Baselines (now often called SB2) began in 2018 as a fork of OpenAI Baselines built on TensorFlow 1.x, created at the U2IS robotics lab (INRIA Flowers team) of ENSTA ParisTech.[1][2] Its successor, Stable-Baselines3 (SB3), is a complete rewrite in PyTorch that reached version 1.0 in March 2021 and is hosted by the GitHub organization of the German Aerospace Center's (DLR) Institute of Robotics and Mechatronics.[3][4][5] Both are released under the MIT License.[1][3] SB2 has been in maintenance mode since 2021, while SB3 remained in active maintenance in November 2026, with version 2.9.2 published on 2026-10-26.[1][6]
The project's stated aim is to make it easier for researchers and industry users to replicate, refine and compare RL ideas against reliable baselines, and to let beginners work with a more advanced toolset "without being buried in implementation details."[3] Its maintainers also caution that the library assumes some prior knowledge of reinforcement learning.[3]
Quick facts
| Attribute | Stable Baselines (SB2) | Stable-Baselines3 (SB3) |
|---|---|---|
| Repository | hill-a/stable-baselines | DLR-RM/stable-baselines3 |
| Repository created | 2018-07-02 | 2020-05-05 |
| First release in changelog | 0.1.6 (2018-07-27) | 0.1.0 pre-release (2020-01-20) |
| Version 1.0 | 2018-08-20 (1.0.0 on PyPI) | 2021-03-15 (changelog date) |
| Last or latest release | 2.10.2 (2021-04-05) | 2.9.2 (2026-10-26) |
| Deep learning backend | TensorFlow 1.x (1.8.0 to 1.14.0) | PyTorch (2.8 or later as of 2.9) |
| Environment API | OpenAI Gym | Gymnasium (since 2.0.0) |
| Status | Maintenance mode | Actively maintained |
| License | MIT | MIT |
| Paper | Cited as a GitHub repository (2018) | JMLR 22(268), 2021 |
Sources: GitHub repository metadata, project changelogs, READMEs and PyPI.[1][2][3][6][7][8][20]
History
Fork of OpenAI Baselines (2018)
OpenAI Baselines, released by OpenAI in 2017, was a set of reference implementations of RL algorithms. The Stable-Baselines3 paper notes that some RL libraries "are notoriously hard to understand," gives OpenAI Baselines as the example, and calls that criticism "a major starting point" for Stable Baselines.[4] The fork's GitHub repository was created on 2018-07-02.[1] Its first changelog entry, release 0.1.6 dated 2018-07-27, was titled "Deobfuscation of the code base + pep8 and fixes": it documented the whole codebase, reformatted it to the PEP8 style, removed duplicated code, fixed TensorFlow session handling and added Atari and logger tests.[2]
Release 1.0.1 on 2018-08-20, titled "Refactored Stable Baselines," moved A2C, ACER, ACKTR, DDPG, DQN, GAIL, TRPO, PPO1 and PPO2 under a single common class and added callbacks, saving and loading, prediction for every algorithm, and automatic detection of the action space.[2] HTML documentation hosted on Read the Docs followed in 1.0.7 (2018-08-29), and version 2.0.0 (2018-09-18) added TensorBoard logging, a registry that lets policies be passed by name as strings, and pre-built Docker images.[2]
The README lists the main differences from OpenAI Baselines as a unified structure for all algorithms, a PEP8-compliant code style, documented functions and classes, more tests and code coverage, and additional algorithms.[1] Later SB2 releases added Soft Actor-Critic (2.4.0, 2019-01-17), a from-scratch reimplementation of Hindsight Experience Replay (2.6.0, 2019-06-12), Twin Delayed DDPG or TD3 (2.7.0, 2019-07-31), and an optional rather than required MPI dependency (2.8.0, 2019-09-29).[2] According to the SB3 paper, the fork and OpenAI Baselines "quickly diverged."[4]
Rewrite in PyTorch (2020 to 2021)
In his announcement of Stable-Baselines3, maintainer Antonin Raffin explained that SB2 still relied on the initial OpenAI Baselines codebase and that, with TensorFlow 2 approaching, more and more of the internal TensorFlow code it used was being deprecated. After discussion with the community, the maintainers chose a complete rewrite in PyTorch, a project first named "torchy-baselines."[5] The SB3 paper adds that PyTorch was the framework preferred by a majority of users in a survey the team ran.[4]
The SB3 changelog's first pre-release, 0.1.0 (2020-01-20), shipped A2C, CEM-RL, PPO, SAC and TD3, working only with Box spaces, together with state-dependent exploration.[6] The DLR-RM/stable-baselines3 repository was created on 2020-05-05, and the first stable-baselines3 package reached PyPI two days later.[3][8] DQN and DDPG arrived in 0.8.0 (2020-08-03), and Hindsight Experience Replay with online and offline sampling in 0.10.0 (2020-10-28).[6][7]
Raffin's blog post presenting version 1.0 is dated 2021-02-28 and says the release followed "several months of beta."[5] The changelog dates release 1.0, labeled "First Major Version," to 2021-03-15; the GitHub release and the PyPI upload are dated 2021-03-17.[6][7][8] The original TensorFlow library received a final release, 2.10.2, on 2021-04-05 with a warning that the package was in maintenance mode and that users should move to SB3, for which a migration guide is provided.[2]
The project is described in "Stable-Baselines3: Reliable Reinforcement Learning Implementations" by Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus and Noah Dormann, published in the Journal of Machine Learning Research (volume 22, paper 268) in November 2021 after submission in December 2020.[4]
Later development (2021 to 2026)
| Version | Date | Main change |
|---|---|---|
| 1.1.0 | 2021-07-01 | Dict observation support, timeout handling, HER refactored into a replay buffer usable by any off-policy algorithm |
| 1.4.0 | 2022-01-18 | TRPO and ARS added to SB3-Contrib, experimental multi-environment training for off-policy algorithms |
| 1.6.0 | 2022-07-11 | Recurrent PPO (PPO with LSTM) added to SB3-Contrib |
| 1.6.2 | 2022-10-10 | Progress bar in learn(); RL Zoo installable as the rl_zoo3 package |
| 2.0.0 | 2023-06-22 | Gymnasium becomes the primary environment backend |
| 2.4.0 | 2024-11-18 | CrossQ added to SB3-Contrib, Gymnasium 1.0 support |
| 2.5.0 | 2025-01-27 | SimBa added to SBX, NumPy 2.0 support |
| 2.7.0 | 2025-07-25 | n-step returns for all off-policy algorithms |
| 2.8.0 | 2026-04-01 | Python 3.9 dropped, Python 3.13 support, documentation moved to Markdown |
| 2.9.0 | 2026-06-15 | PyTorch 2.8 minimum, pandas and matplotlib made optional |
| 2.9.2 | 2026-10-26 | Bug fixes; last release to support Python 3.10 |
Source: SB3 changelog.[6]
Version 2.0.0 made Gymnasium, the successor to OpenAI Gym, SB3's primary environment backend, while older Gym versions 0.21 and 0.26 remained usable through the shimmy compatibility package.[6] The README states that, since most of the original roadmap has been implemented, no major changes are planned for the core library, which now concentrates on bug fixes and maintenance; newer algorithms go to SB3-Contrib, faster variants to SBX, and the RL Zoo keeps its own roadmap.[3]
Design
The SB3 paper sets out the project's priorities. To keep the library simple to use and maintain, it covers model-free, single-agent algorithms and leaves imitation learning and offline RL to external projects (the imitation library and d3rlpy). It favors stable implementations over new features or algorithms, tries to avoid breaking changes, and offers a documented API modeled on scikit-learn. The code favors readability and simplicity over modularity, while using object-oriented programming to limit duplication.[4]
A typical training script is a few lines long. The README example trains PPO on the CartPole task:[3]
from stable_baselines3 import PPO
model = PPO("MlpPolicy", "CartPole-v1").learn(10_000)
Library features described in the paper and README include:[3][4]
- logging to CSV files and TensorBoard, with custom metrics
- user-defined callbacks that can monitor or modify training
- parallel ("vectorized") environments to speed up data collection
- common environment wrappers, such as Atari preprocessing matching the original DQN experiments
Dictobservation spaces, custom environments and custom policies- type hints on all functions, checked by static type checking
- an environment checker that validates custom environments
The paper reports that automated unit tests cover 95% of the code, that continuous integration checks tests, types, code style and documentation on every change, and that each algorithm was checked against published results by comparing learning curves.[4] Raffin's announcement gives examples of the implementation details this involved: PyTorch's RMSprop optimizer differs from TensorFlow's, so SB3 ships a TensorFlow-like variant, and two otherwise identical A2C runs that differed only in the optimizer's epsilon value performed differently.[5]
SB3 has no built-in tools for converting models to other frameworks, but its documentation explains how to export the policy network, including to ONNX.[9]
Algorithms
SB2 inherited most of its algorithms from OpenAI Baselines and added SAC and TD3; its HER implementation was rewritten from scratch.[1] SB3 trimmed the core library to seven algorithms and moved newer or less mature methods into companion packages.[3][4]
| Algorithm | SB2 | SB3 core | SB3-Contrib | SBX |
|---|---|---|---|---|
| A2C | Yes | Yes | ||
| ACER | Yes | |||
| ACKTR | Yes | |||
| ARS | Yes | |||
| CrossQ | Yes | Yes | ||
| DDPG | Yes | Yes | Yes | |
| DQN | Yes | Yes | Yes | |
| DroQ | Yes | |||
| GAIL | Yes | |||
| HER | Yes | Yes | ||
| Maskable PPO | Yes | |||
| PPO | Yes (PPO1, PPO2) | Yes | Yes | |
| QR-DQN | Yes | |||
| Recurrent PPO | Yes | |||
| SAC | Yes | Yes | Yes | |
| SAC-N | Yes | |||
| SimBa | Yes | |||
| TD3 | Yes | Yes | Yes | |
| TQC | Yes | Yes | ||
| TRPO | Yes | Yes |
Sources: SB2 README, SB3 README and algorithm table, SBX README.[1][3][10][11]
The algorithms span on-policy actor-critic and policy-gradient methods (A2C, PPO, TRPO), off-policy methods for continuous control (DDPG, TD3, SAC, TQC), value-based methods for discrete actions (DQN, QR-DQN), and the goal-conditioned replay strategy HER.[3][4] In the SB3 implementation, HER is a replay buffer class (HerReplayBuffer) passed to an off-policy algorithm rather than a standalone algorithm.[6]
SB3 also carries generalized state-dependent exploration (gSDE), a method by Raffin, Jens Kober and Freek Stulp that replaces the step-by-step noise of standard deep RL exploration, which produces jerky motion on real robots, with smoother state-dependent noise. Their paper evaluated it in PyBullet simulation and on three physical robots (a tendon-driven elastic robot, a quadruped and an RC car) and released its code in the SB3 repository.[12] State-dependent exploration for A2C, PPO, SAC and TD3 was already part of SB3's first pre-release.[6]
Ecosystem
RL Baselines Zoo
RL Baselines Zoo, started by Raffin for SB2 and continued as RL Baselines3 Zoo for SB3, is a training framework with scripts to train and evaluate agents, tune hyperparameters, plot results and record videos. It includes tuned hyperparameters for common environments and pretrained agents trained with them.[4][13] The SB3 paper says these hyperparameters cover simple control tasks, PyBullet environments and Atari games, and were optimized with Optuna.[4] Since SB3 1.6.2 (October 2022) the Zoo can be installed as the rl_zoo3 package.[6]
SB3-Contrib
SB3-Contrib (sb3-contrib), whose repository was created in September 2020 under the Stable-Baselines-Team GitHub organization, holds algorithms and tools considered experimental, such as implementations of recent papers.[10] According to the SB3 paper, contrib code does not have to be tightly integrated with the main codebase but goes through the same review requirements, and implementations that "have stood the test of time" may move into the main repository.[4]
Stable Baselines Jax (SBX)
SBX, published on PyPI as sbx-rl, is described by its author as a "proof of concept version of Stable-Baselines3 in Jax." It follows the SB3 API and implements recent algorithms such as DroQ, CrossQ and SimBa alongside SAC, PPO and others.[11] The SB3 documentation says SBX offers fewer features than SB3 but can be much faster, citing speedups of "up to 20x," and that it works with the RL Zoo.[11][14]
Integrations
The SB3 documentation describes integrations with Weights & Biases (a callback for experiment tracking) and the Hugging Face Hub, where official pretrained models are kept under the sb3 organization and the RL Zoo can push and load models directly.[15] The README also points to detailed training logs and reports on the Open RL Benchmark platform.[3]
Adoption and reception
The SB3 paper reported that at the time of writing SB2 had more than 650 closed issues, 220 merged pull requests and 200 citations on Google Scholar, and that SB3 had more than 800 GitHub stars.[4] As of November 2026, the SB3 repository had about 14,000 stars and the SB2 repository about 4,300.[1][3] Applications named in the paper range from research baselines to continual learning when attacking WiFi networks and dampening bridge vibrations.[4]
SB3 is used in teaching material and in robotics tooling. The first hands-on unit of the Hugging Face Deep Reinforcement Learning Course trains a PPO agent on the LunarLander environment with Stable-Baselines3.[16] The Isaac Lab documentation uses its Stable-Baselines3 vectorized environment wrapper in the tutorial on training a policy to balance a cartpole.[17] The DLR Institute of Robotics and Mechatronics lists SB3 among its published software.[18]
The SB3 paper compares the library with several others on documentation, testing and activity. In the authors' table, SB3 has 95% test coverage against 49% for OpenAI Baselines, 94% for Tianshou, 74% for Acme and 81% for Tensorforce. The authors say RLlib also scores highly but targets a different use case, prioritizing scalability, distributed training and multi-agent support at the cost of a larger and more complex codebase.[4]
The authors of CleanRL, a library of single-file RL implementations, argued that modular libraries such as SB3 make an algorithm's implementation details hard to follow. Their paper notes that stepping through a PPO run on Atari with SB3 in a debugger involves moving between 20 Python files containing more than 4,000 lines of code. The same paper describes SB3 as a good fit for users who want to apply PPO without deep RL expertise, and proposes prototyping new techniques in CleanRL and porting them to SB3 for wider adoption.[19]
People and funding
SB2 was maintained by Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave and Anssi Kanervisto.[1] As of November 2026, SB3 lists the same five plus Quentin Gallouédec as maintainers.[3] The SB3 paper's author affiliations were the DLR Robotics and Mechatronics Center (Raffin and Dormann), CEA at University Paris-Saclay (Hill), the University of California, Berkeley (Gleave), the University of Eastern Finland (Kanervisto) and Kiteswarms GmbH (Ernestus).[4] The initial development of SB3 was partly funded by the Helmholtz project "Reduced Complexity Models" and by the EU Horizon 2020 programme under grant 951992 (VeriDream).[3]
Continue reading
- Proximal Policy Optimization (PPO)Proximal Policy Optimization (PPO) is an on-policy policy gradient reinforcement learning algorithm that stabilizes training by clipping the policy update so the new policy stays close ("proximal") to the old…Mentioned 13 times above
- OpenAI BaselinesOpenAI Baselines is a collection of open-source, high-quality reference implementations of reinforcement learning (RL) algorithms released by OpenAI.Mentioned 9 times above
- Gym (OpenAI Gym / Gymnasium)Gym, often written as OpenAI Gym, is an open source Python toolkit for developing and comparing reinforcement learning algorithms, originally released by openai on April 27, 2016.Mentioned 4 times above
- Soft Actor-CriticSoft Actor-Critic (SAC) is an off-policy, maximum-entropy deep reinforcement learning algorithm that trains a stochastic actor-critic to maximize expected reward plus the entropy of its own policy, so the…Listed in See also
- Spinning UpSpinning Up in Deep RL is a free, open-source educational resource produced by OpenAI to make deep reinforcement learning (deep RL) easier to learn.Listed in See also
- Robot learningRobot learning studies how robots acquire or improve behavior from data and experience.Listed in See also
More to read
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Stable Baselines team. "hill-a/stable-baselines: A fork of OpenAI Baselines, implementations of reinforcement learning algorithms". GitHub. github.com/...stable-baselines
- ^1 ^2 ^3 ^4 ^5 ^6 ^7Stable Baselines team. "Changelog". Stable Baselines documentation (GitHub source), last release 2021-04-05. github.com/...changelog.rst
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16DLR-RM. "DLR-RM/stable-baselines3: PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms". GitHub, accessed November 2026. github.com/...stable-baselines3
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17Raffin, Antonin; Hill, Ashley; Gleave, Adam; Kanervisto, Anssi; Ernestus, Maximilian; Dormann, Noah. "Stable-Baselines3: Reliable Reinforcement Learning Implementations". Journal of Machine Learning Research 22(268):1-8, 2021. jmlr.org/...20-1364
- ^1 ^2 ^3 ^4Raffin, Antonin. "Stable-Baselines3: Reliable Reinforcement Learning Implementations". araffin.github.io, 2021-02-28. araffin.github.io/...sb3
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Stable-Baselines3 developers. "Changelog". Stable-Baselines3 documentation, accessed November 2026. stable-baselines3.readthedocs.io/...changelog
- ^1 ^2 ^3DLR-RM. "Releases: DLR-RM/stable-baselines3". GitHub. github.com/...releases
- ^1 ^2 ^3Python Package Index. "stable-baselines3". PyPI, accessed November 2026. pypi.org/...stable-baselines3
- ^Stable-Baselines3 developers. "Exporting models". Stable-Baselines3 documentation. stable-baselines3.readthedocs.io/...export
- ^1 ^2Stable-Baselines-Team. "stable-baselines3-contrib: Contrib package for Stable-Baselines3 - Experimental reinforcement learning (RL) code". GitHub. github.com/...stable-baselines3-contrib
- ^1 ^2 ^3Raffin, Antonin. "araffin/sbx: SBX: Stable Baselines Jax (SB3 + Jax) RL algorithms". GitHub. github.com/...sbx
- ^Raffin, Antonin; Kober, Jens; Stulp, Freek. "Smooth Exploration for Robotic Reinforcement Learning". Proceedings of the 5th Conference on Robot Learning, PMLR 164:1634-1644, 2022 (arXiv:2005.05719). arxiv.org/...2005.05719
- ^DLR-RM. "rl-baselines3-zoo: A training framework for Stable Baselines3 reinforcement learning agents, with hyperparameter optimization and pre-trained agents included". GitHub. github.com/...rl-baselines3-zoo
- ^Stable-Baselines3 developers. "Stable Baselines Jax (SBX)". Stable-Baselines3 documentation. stable-baselines3.readthedocs.io/...sbx
- ^Stable-Baselines3 developers. "Integrations". Stable-Baselines3 documentation. stable-baselines3.readthedocs.io/...integrations
- ^Hugging Face. "Train your first Deep Reinforcement Learning Agent". Hugging Face Deep RL Course, Unit 1. huggingface.co/...hands-on
- ^Isaac Lab. "Training with an RL Agent". Isaac Lab Documentation. isaac-sim.github.io/...run_rl_training
- ^German Aerospace Center (DLR), Institute of Robotics and Mechatronics. "Stable Baselines3". DLR. dlr.de/...stable-baselines3
- ^Huang, Shengyi; Dossa, Rousslan Fernand Julien; Ye, Chang; Braga, Jeff; Chakraborty, Dipam; Mehta, Kinal; Araújo, João G.M. "CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms". Journal of Machine Learning Research 23(274):1-18, 2022. jmlr.org/...21-1342
- ^Python Package Index. "stable-baselines". PyPI. pypi.org/...stable-baselines
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
2 revisions · v2 · 2,802 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check before publication on November 18, 2026: 112 claims checked against 24 cited and independent sources; 0 material and 2 minor corrections applied before publication.
Cite this page: AI Wiki. "Stable Baselines." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/stable_baselines
