Episode (Reinforcement Learning)

An episode in reinforcement learning is one complete sequence of interaction between an agent and its environment, starting from an initial state and ending when a terminal state is reached. It produces a trajectory of states, actions, and rewards, and the agent's goal is to maximize the return, the cumulative (optionally discounted) reward accumulated over that episode ^[1]. Episodes are the basic unit of experience in episodic tasks: when one ends, the environment resets and a new episode begins independently of how the previous one finished ^[1].

The term is also called a rollout or a trajectory, and tasks that break naturally into episodes (such as games or maze runs) are called episodic tasks, as distinct from continuing tasks that go on indefinitely without a terminal state ^[1].

ELI5 (Explain like I'm 5)

Imagine you are playing a video game. You start the level, run around collecting coins, dodge some enemies, and eventually either win the level or lose all your lives. That whole attempt, from start to finish, is one episode. When the level ends, you get to try again from the beginning, but this time you remember what worked and what didn't. Each time you play through the level is a new episode, and you get a little better each time because you learn from your past attempts.

In reinforcement learning, a computer agent does the same thing. It starts in some situation, makes a bunch of decisions, gets feedback (rewards or penalties), and eventually the round ends. Then it starts over, using what it learned to make smarter decisions next time.

What is an episode in reinforcement learning?

An episode is a finite sequence of agent-environment interaction that runs from a starting state until a terminal state is reached, at which point the environment resets ^[1]. In the framework of Markov decision processes (MDPs), an episode is the finite sequence of transitions:

$S_0, A_0, R_1, S_1, A_1, R_2, \ldots, S_{T-1}, A_{T-1}, R_T, S_T$

where:

• $S_0$ is the initial state, sampled from a start-state distribution $\rho_0$
• $A_t$ is the action taken by the agent at time step $t$, chosen according to its policy $\pi(a \mid s)$
• $R_{t+1}$ is the reward received after taking action $A_t$ in state $S_t$
• $S_T$ is the terminal state, at which point the episode ends
• $T$ is the length of the episode (which may vary across episodes)

The complete sequence is also referred to as a trajectory, denoted $\tau = (S_0, A_0, S_1, A_1, \ldots, S_T)$. Sutton and Barto formalize this in Reinforcement Learning: An Introduction by defining episodic tasks as problems where the agent-environment interaction breaks naturally into subsequences, each of which ends in a special terminal state. In their words, "Each episode ends in a special state called the terminal state, followed by a reset to a standard starting state or to a sample from a standard distribution of starting states" ^[1]. A key property is that the start of each episode is independent of how the previous one ended ^[1].

Structure of an episode

Every episode consists of several core components that define the agent's interaction with the environment:

Scroll sideways for more →

The environment is typically reset after each episode, and the agent begins a new episode from a fresh initial state. This reset mechanism is what distinguishes episodic tasks from continuing tasks.

What is the return over an episode?

The return $G_t$ is the cumulative reward the agent seeks to maximize, calculated from time step $t$ to the end of the episode ^[1]. There are two standard formulations:

Finite-horizon undiscounted return:

$G_t = R_{t+1} + R_{t+2} + R_{t+3} + \cdots + R_T = \sum_{k=0}^{T-t-1} R_{t+k+1}$

This simply sums all future rewards until the episode ends. It is well-defined in episodic tasks because $T$ is finite ^[1].

Discounted return:

$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots = \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}$

where $\gamma \in [0, 1)$ is the discount factor. The discount factor controls how much the agent values future rewards relative to immediate ones:

Scroll sideways for more →

In episodic tasks, setting $\gamma = 1$ is sometimes appropriate because the finite episode length guarantees that the return is bounded. In continuing tasks, a discount factor strictly less than 1 is necessary to keep the infinite sum from diverging ^[1].

What is the difference between episodic and continuing tasks?

Reinforcement learning problems fall into two broad categories based on whether the interaction has natural endpoints. In an episodic task the agent-environment interaction "breaks naturally into subsequences" that each end in a terminal state, whereas in a continuing task the interaction "goes on continually without limit" ^[1].

Scroll sideways for more →

Sutton and Barto introduce a unified notation by defining a special absorbing state that transitions only to itself and generates zero reward. This allows the finite sum in episodic returns to be expressed as an infinite sum, unifying the notation for both task types, because the sum of rewards is the same with or without the added absorbing state ^[1].

How does an episode end: termination vs. truncation?

An episode can end in two distinct ways, and the distinction has practical consequences for learning algorithms.

Termination occurs when the agent reaches a genuine terminal state as defined by the MDP. Examples include winning or losing a game, a robot completing its task, or reaching a goal location. When an episode terminates, the future value from that state is truly zero.

Truncation occurs when an episode is cut short by an external constraint that is not part of the MDP, such as a maximum time step limit imposed for practical reasons. In this case, the episode ends, but the agent has not reached a true terminal state; there would have been additional rewards to collect.

This distinction matters for value estimation. Pardo et al. (2018), in their paper "Time Limits in Reinforcement Learning," show that conflating truncation with termination leads to incorrect value estimates. They argue that when time limits are used only to diversify training experience, "this insight should be incorporated by bootstrapping from the value of the state at the end of each partial episode" ^[4]. Treating truncated transitions as if they were true terminations degraded performance by roughly 20-40% on standard MuJoCo continuous-control benchmarks in their experiments ^[4].

Scroll sideways for more →

The Gymnasium library (the maintained fork of OpenAI Gym) addressed this by replacing the single done flag with separate terminated and truncated boolean values in its step() API, a change introduced in version 0.26 ^[8]. As the Farama Foundation documentation explains, the API "was changed in version 0.26 ... removing done in favor of terminated and truncated," because the distinction "is critical for reinforcement learning bootstrapping algorithms": when an episode ends due to termination the agent does not bootstrap, and when it ends due to truncation it does ^[8].

Related terminology

Several terms in reinforcement learning are closely related to the concept of an episode and are sometimes used interchangeably, though there are subtle distinctions.

Scroll sideways for more →

How are episodes used in learning algorithms?

Episodes play different roles depending on the class of reinforcement learning algorithm being used.

Monte Carlo methods

Monte Carlo methods have the strongest dependence on episodes. These methods estimate value functions by averaging the actual returns observed across many complete episodes. Because they use the full return $G_t$ rather than bootstrapping from intermediate estimates, Monte Carlo methods require episodes to terminate before any learning updates can occur ^[1].

The value of a state $s$ under policy $\pi$ is estimated as:

$V(s) \approx \frac{1}{N(s)} \sum_{i=1}^{N(s)} G_t^{(i)}$

where $N(s)$ is the number of times state $s$ was visited across all episodes, and $G_t^{(i)}$ is the return observed from the $i$-th visit to state $s$.

There are two variants: first-visit Monte Carlo, which counts only the first time $s$ appears in each episode, and every-visit Monte Carlo, which counts every occurrence ^[1].

Temporal difference learning

Temporal difference (TD) learning methods can update value estimates at every time step within an episode, without waiting for the episode to end. TD methods use bootstrapping, updating estimates based on other estimates:

$V(S_t) \leftarrow V(S_t) + \alpha \left[ R_{t+1} + \gamma V(S_{t+1}) - V(S_t) \right]$

Because TD methods do not need to wait for episode completion, they can also be applied to continuing tasks. However, in episodic tasks, episode boundaries still serve an important role by defining when the environment resets and providing natural evaluation points.

Policy gradient methods

The REINFORCE algorithm, introduced by Williams (1992), is a Monte Carlo policy gradient method that updates policy parameters after each complete episode ^[3]. The policy gradient is estimated as:

$\nabla J(\theta) \approx \sum_{t=0}^{T-1} \nabla \log \pi_\theta(A_t | S_t) \cdot G_t$

Because REINFORCE uses the full episode return $G_t$, it must wait for the episode to finish before updating. This episode-level update leads to high variance in gradient estimates, which is a known limitation of the algorithm. Actor-critic methods address this by using TD-style bootstrapping to reduce variance while still operating within the episodic framework.

Deep Q-Networks and experience replay

In Deep Q-Network (DQN) algorithms, individual transitions $(s_t, a_t, r_{t+1}, s_{t+1})$ from many different episodes are stored in a replay buffer. During training, random minibatches of transitions are sampled from this buffer, breaking the temporal correlations between consecutive time steps within an episode. This technique, called experience replay, was a key innovation in the original DQN paper by Mnih et al. (2015), which reported human-level performance across 49 Atari 2600 games ^[2].

While experience replay decouples learning from the sequential structure of individual episodes, episodes still serve as the organizing unit for data collection: the agent runs episodes to fill the replay buffer.

Episode design considerations

The design of episodes, including their length, initial state distribution, and termination conditions, significantly affects learning performance.

Episode length

Episode length influences the tradeoff between exploration and exploitation:

• Short episodes allow the agent to attempt more episodes within a fixed training budget, promoting broader exploration of different initial states and early trajectories. However, they may prevent the agent from learning long-horizon behaviors.
• Long episodes give the agent more time to discover long-term strategies but reduce the number of episodes the agent can experience, potentially slowing early learning.

Many practical implementations use a maximum episode length (time limit) to prevent episodes from running indefinitely, especially in environments where the agent might get stuck. This time limit is the truncation case discussed above, and Pardo et al. (2018) showed it should be handled with bootstrapping rather than treated as a true terminal state ^[4].

Initial state distribution

Randomizing the initial state across episodes helps the agent learn a more general policy. If the agent always starts from the same state, it may overfit to that specific starting configuration. The Gymnasium reset() method supports seeded randomization of initial states for reproducibility ^[8].

Curriculum learning

In curriculum learning, the difficulty of episodes is gradually increased over training. For example, a robot learning to walk might begin with short, easy episodes on flat terrain and progress to longer episodes on rough terrain. This staged approach can accelerate convergence compared to training on the full-difficulty task from the start.

How do you implement an episode loop in code?

In modern RL frameworks such as Gymnasium (the maintained fork of OpenAI Gym), episodes are managed through a standardized API.

The episode loop

A typical episode loop follows this pattern:

1. Call env.reset() to initialize the environment and get the initial observation
2. Repeat:
   • Select an action based on the current observation and policy
   • Call env.step(action) to execute the action
   • Receive the next observation, reward, terminated flag, truncated flag, and info dict
   • Store the transition for learning
   • If terminated or truncated, break out of the loop
3. Update the agent's policy using the collected experience
4. Repeat from step 1 for the next episode

In the Gymnasium migration from the old API, code that previously checked a single done flag is updated to done = terminated or truncated ^[8].

Episode tracking and logging

During training, key per-episode metrics are typically logged:

Scroll sideways for more →

These metrics, tracked over hundreds or thousands of episodes, form the learning curve that indicates whether the agent is improving.

Examples of episodes across domains

The concept of an episode manifests differently depending on the application domain.

Scroll sideways for more →

Historical context

The concept of an episode in reinforcement learning has roots in the early work on dynamic programming and optimal control theory. Richard Bellman's work on sequential decision processes in the 1950s laid the groundwork by formalizing finite-horizon decision problems ^[9]. The term "episode" became standard in the RL literature through Sutton and Barto's influential textbook Reinforcement Learning: An Introduction (first edition 1998, second edition 2018), which systematically distinguished episodic tasks from continuing tasks ^[1].

The practical significance of episodes grew with the development of Monte Carlo methods for RL and later with policy gradient methods like REINFORCE (Williams, 1992) ^[3]. The advent of deep reinforcement learning, particularly DQN (Mnih et al., 2013, 2015) and AlphaGo (Silver et al., 2016), brought renewed attention to episode design and management as agents were trained over millions of episodes on complex tasks ^[2][5][6].

See also

• Reinforcement learning
• Markov decision process
• Return
• Reward
• Discount factor
• Policy
• Policy gradient
• Temporal-difference learning
• Deep Q-Network
• Experience replay

References

1. Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction* (2nd ed.). MIT Press. http://incompleteideas.net/book/the-book-2nd.html ↩
2. Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2015). Human-level control through deep reinforcement learning. *Nature*, 518(7540), 529-533. ↩
3. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. *Machine Learning*, 8(3-4), 229-256. ↩
4. Pardo, F., Tavakoli, A., Levdik, V., & Kormushev, P. (2018). Time limits in reinforcement learning. *Proceedings of the 35th International Conference on Machine Learning (ICML)*. https://arxiv.org/abs/1712.00378 ↩
5. Silver, D., Huang, A., Maddison, C. J., et al. (2016). Mastering the game of Go with deep neural networks and tree search. *Nature*, 529(7587), 484-489. ↩
6. Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2013). Playing Atari with deep reinforcement learning. *arXiv preprint arXiv:1312.5602*. ↩
7. OpenAI. (2016). OpenAI Gym. *arXiv preprint arXiv:1606.01540*.
8. Towers, M., Terry, J. K., Kwiatkowski, A., et al. (2023). Gymnasium. Farama Foundation. https://gymnasium.farama.org/ ↩
9. Bellman, R. (1957). *Dynamic Programming*. Princeton University Press. ↩
10. Puterman, M. L. (2014). *Markov Decision Processes: Discrete Stochastic Dynamic Programming*. John Wiley & Sons.
11. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. *arXiv preprint arXiv:1707.06347*.
12. Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. *Proceedings of the 35th International Conference on Machine Learning (ICML)*.