RAGEN-2

RAGEN-2 is a 2026 research paper and public code extension for diagnosing and mitigating reasoning collapse during reinforcement learning of multi-turn large language model agents. The paper defines template collapse as a regime in which generated reasoning remains varied for a given input but becomes weakly dependent on which input produced it. Because ordinary reasoning entropy can remain high in that regime, the authors propose in-batch mutual-information proxies and a reward-variance filter for training updates.^[1]

The paper was accepted for an oral presentation at the International Conference on Machine Learning (ICML 2026), whose program also lists a companion poster. RAGEN-2 is not a separately released model or hosted agentic AI service. It is a paper, diagnostic implementation, and set of filtering changes within the existing RAGEN research repository.^[2][3][7]

The authors report positive average filtering deltas across their main model and algorithm rows, but individual task cells include negative results. Their experiments are single-agent and use controlled research environments. The paper also states that reward variance becomes a weaker signal under heavy environment randomness and that aggressive filtering can reduce exploration. These boundaries matter because the method selects prompts with a useful sampled reward spread; it does not make an uninformative reward function informative.^[1]

Publication and relationship to RAGEN

Zihan Wang and fifteen coauthors submitted the RAGEN-2 preprint to arXiv on April 7, 2026. The paper lists affiliations at Northwestern University, UIUC, Imperial College London, Oxford University, the University of Washington, Microsoft, Stanford University, and an independent affiliation. Microsoft Research catalogs the paper as an ICML 2026 publication, while the conference program identifies Zihan Wang as Zihan "Zenus" Wang and records both oral and poster events.^[1][2][3][4]

RAGEN-2 continues the project introduced by the 2025 RAGEN paper. That earlier work presented StarPO, or State-Thinking-Actions-Reward Policy Optimization, as a framework for optimizing complete interaction trajectories rather than isolated prompt-response pairs. It also introduced the RAGEN software for generating rollouts, assigning rewards, and updating policies in interactive environments. RAGEN-2 uses that infrastructure to study whether an agent's reasoning still responds to its current context during training.^[5][7]

The dates attached to the public artifacts refer to different events. The repository update log labels March 12, 2026 as the RAGEN-2 release. The academic preprint followed on April 7, and ICML presented the work in July. No tagged GitHub release was visible through the article's August 21 research cutoff, so the March date is best read as the maintainers' code-update label rather than a versioned archival release.^{[1][2][7][13]}

Scroll sideways for more →

Template collapse and the four regimes

RAGEN-2 treats each reasoning turn as a relation between context and generated reasoning. The paper writes the context as X: the system prompt, observations so far, previous actions, and earlier reasoning tokens. It writes the current reasoning sequence as Z, excluding the executable action and boundary tokens. An information-theoretic identity then separates marginal reasoning diversity into input dependence and within-input variation:

H(Z) = I(X; Z) + H(Z | X)

Here, H(Z | X) measures how much reasoning varies when the input is held fixed. I(X; Z) measures how distinguishable the reasoning distributions are across inputs. The paper's point is not that entropy is computed incorrectly. It is that a within-input diversity statistic does not answer whether the output changes coherently when the context changes.^[1]

Scroll sideways for more →

These axes do not measure correctness by themselves. A trace may be highly specific to one input and still contain a wrong plan. They also do not prove that the written reasoning caused the selected action. An independent technical critique by Qinglin Hou makes the same distinction: retrieval of the source prompt tests specificity, not correctness or causal faithfulness.^[1][15]

Mutual-information proxy family

Exact mutual information is not tractable for long token sequences, so the paper estimates input dependence by cross-scoring a training batch. For P prompts and G sampled reasoning traces per prompt, it teacher-forces every trace under every prompt. This produces a matrix of log likelihoods. A matched score evaluates a trace under its true source prompt; a marginal score approximates its likelihood under a uniform mixture of the prompts in that batch.^[1]

The most interpretable proxy is retrieval accuracy. For each trace, the system selects the prompt under which it has the highest likelihood and checks whether that prompt was the source. Under the paper's idealized template-collapse case, accuracy approaches chance, 1/P. Continuous variants average the matched-minus-marginal gap, use either token-normalized or sequence-level scores, and optionally divide by a batch or exponential-moving-average standard deviation. The implementation exposes first-turn and trajectory-sampled views.^[1][9]

The proxy does not need a separate judge model. That does not make it free. The released detector constructs the cross-prompt batches and calls the actor's log-probability computation for them. It can run at a configured frequency rather than every step. Its cost therefore depends on the number of prompts, sampled traces, sequence lengths, and how often cross-scoring is enabled.^[9]

Across the paper's intervention sweeps, trajectory MI-ZScore had a Spearman correlation of +0.39 with task success. The reported reasoning-entropy correlations ranged from -0.11 to -0.14. Those values describe the authors' run collection; they are not universal thresholds for detecting a faulty agent. They also combine runs generated by changes to entropy regularization, KL divergence, and filtering strength, so the correlation does not by itself establish that raising the proxy will improve an arbitrary task.^[1]

Signal-to-noise account and filtering

For each prompt, RAGEN-2 samples several trajectories and computes the variance of their returns. If all returns are nearly equal, centering them into advantages leaves little contrast between trajectories. The paper's gradient analysis reports that task-gradient norm increased across reward-variance buckets while the gradient contribution from KL and entropy regularization stayed roughly flat. The authors interpret low-variance prompts as a regime where task-specific signal weakens and input-agnostic regularization can dominate the update.^[1]

SNR-Aware Filtering applies that account before the parameter update. It ranks prompts by within-prompt reward variance and trains on a selected high-variance subset. In the paper's top-p formulation, the subset is the smallest prefix whose cumulative raw variance reaches fraction rho of total variance mass. A setting of rho=0.9 therefore does not necessarily keep 90 percent of prompts. If a few groups hold most of the variance, the kept prompt count can be much smaller.^[1]

The paper's detailed setup later paraphrases rho=0.9 as "keeping the top 90% of trajectory groups," which conflicts with the formal cumulative-mass equation. The equation is the unambiguous method definition; a reproduction should state which interpretation and code mode it used.^[1]

The public artifacts use two names for this filter. The paper and ICML abstract say SNR-Aware Filtering. The project page, repository README, and current rollout guide say SNR-Adaptive Filtering for the reward-variance implementation. The sources describe the same filter family, but the naming difference should be preserved rather than silently converted into a second algorithm.^[1][2][6][8]

The current repository supports more than one selection rule. Its guide documents raw-score and softmax top-p modes, fractional or absolute top-k selection, minimum thresholds, exclusion of zero-variance groups, and reverse low-variance selection for ablations. The guide's recommended general configuration uses linear score aggregation and excludes zero-variance groups. By contrast, the repository's main-table reproduction guide says those scripts use softmax top-p and include zero-variance groups. The published paper formalizes cumulative raw reward-variance mass. These configurations are related but not identical, so reproduction requires choosing the paper or code snapshot and recording the exact options.^[1][8][10]

That reproduction guide also defaults to five tasks by adding WebShop to the four columns in manuscript Table 4. It is therefore a current experiment launcher, not a byte-for-byte record of the published table scope.^[1][10]

Reward variance also has more than one semantic cause. Uniformly high rewards may mean a prompt is already easy, while uniformly low rewards may mean it is too hard or that the reward fails to distinguish useful behavior. Filtering can omit either case. It does not repair missing credit assignment or a poorly specified verifier. Hou characterizes the intervention as close to curriculum or self-paced selection and argues that the diagnostic contribution is stronger than the paper's mechanistic explanation.^[15]

Experimental design

The paper describes a seven-environment testbed. Its main four-task matrix covers Sokoban, FrozenLake, MetaMathQA, and Countdown. SearchQA, WebShop, and DeepCoder extend the scope to search, simulated shopping, and code generation.^[1]

Scroll sideways for more →

The two MetaMathQA interaction descriptions above come from different parts of the paper. Table 3 and the environment overview call the task multi-turn and allow revisions, but the detailed central setup specifies one turn and one action. This is an internal reporting inconsistency; the article does not infer that multiple retries occurred in those central runs.^[1]

The central setup trains Qwen2.5-3B with the veRL/HybridFlow stack for up to 400 rollout-update iterations. It compares PPO, DAPO, GRPO, and Dr. GRPO. Each iteration collects 128 trajectories from eight prompts with sixteen trajectories per prompt. Evaluation uses 512 fixed validation prompts per environment with sampling temperature 0.5. The four-task experiments use five interaction turns for Sokoban and FrozenLake and one turn for MetaMathQA and Countdown.^[1]

The paper reports an actor learning rate of 1e-6, a critic rate of 1e-5, generalized advantage estimation with gamma and lambda both set to 1.0, an entropy coefficient of 0.001, and asymmetric PPO clipping at 0.2 and 0.28. It says the experiments ran on NVIDIA GPUs but does not give the model and count for the full result matrix. The current scripts include multi-H100 examples, which are reproduction instructions rather than evidence of the exact original hardware.^[1][10]

Reported results

The main result table reports the unfiltered peak success-rate average followed by the filtering delta. The two Qwen2.5-VL rows cover only Sokoban and FrozenLake; the other rows average the four central tasks.^[1]

Scroll sideways for more →

Every row has a positive average delta, but several task cells do not. The negative deltas are GRPO on FrozenLake (-3.0), Dr. GRPO on Sokoban (-0.4), Qwen2.5-0.5B on MetaMathQA (-0.2) and Countdown (-0.7), Qwen2.5-7B on FrozenLake (-0.6), Qwen2.5-3B-Instruct on Countdown (-0.6), and Llama 3.2-3B on FrozenLake (-0.2) and Countdown (-1.2). The paper's evidence therefore supports average gains across the listed rows, not improvement in every tested combination.^[1]

The official project page displays +2.3 for the GRPO FrozenLake delta, while Table 4 in the manuscript reports -3.0. The result discussion here follows the manuscript.^[1][6]

The fixed-rollout-budget sweep holds the total at 128 trajectories while changing prompts per batch and trajectories per prompt. The authors report that computing reward variance took less than 0.1 percent of iteration time. Once filtering removed groups from the gradient update, step time fell by 26 to 41 percent in the listed Sokoban configurations, with VRAM changes within 0.44 GB. This comparison keeps rollout count fixed but reduces the number of groups entering backpropagation, so it is not an equal-update-data comparison.^[1]

Some ablations show where the filter can fail. In a high- to low-reward-variance quartile experiment, the MI proxy fell from 0.95 to 0.73 and task success fell from 21.1 percent to about 11 percent. The four printed success values are 21.1, 19.5, 10.7, and 11.0, so that sequence itself is not strictly monotonic even though the paper describes a monotonic decline. A separate FrozenLake GRPO setting had a -5.0-point filtering delta and a reward-variance standard-deviation-to-mean ratio of 0.33. In the environment-noise study, the filter's advantage closed at 80 to 100 percent stochasticity.^[1]

Software coverage and licensing

The inspected repository snapshot is commit 20daedc47558e000f7de912b060646bf2e8026bd, which was the tip of main at the research cutoff. It includes the RAGEN-2 PDF, filtering and MI code, experiment launchers, configs, tests, and guides. The repository also pins gitlinks for veRL, a minimal WebShop implementation, a Lean server, and a spatial-environment dependency. The arXiv manuscript is distributed under CC BY 4.0. The repository code is released under the MIT License, including its warranty disclaimer; datasets, base models, submodules, and external environments keep their own licenses.^[1][7][12]

Three environment counts describe different scopes. The paper's Table 3 reports seven testbed environments. A separate reasoning-length figure is captioned "across eight environments," but its eight panels include two Sokoban configurations and tasks outside Table 3, so it is not a second clean enumeration of the testbed. The README advertises ten built-in environments: Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, Lean, Bandit, Countdown, MetaMathQA, and Sudoku. The setup script names six base environments and an optional search installation that downloads about 87 GB. Software presence, setup support, and inclusion in a published experiment should not be treated as the same claim.^[1][7][11]

The setup script creates a Python 3.12 environment, initializes submodules, and says it was validated on NVIDIA H100, H200, and B200 GPUs. Some dependency versions are pinned or bounded, but many packages in the Python setup remain unpinned. The repository provides task-specific reproduction documents, yet a generic clone of the moving main branch does not identify the same dependency state as the commit-pinned snapshot.^[7][10][11]

No tagged release or trained RAGEN-2 checkpoint was identified in the inspected public artifacts. That does not prevent reproducing training from the supplied scripts, but it means the public release is source and configuration material rather than a frozen set of paper checkpoints. The paper also does not report random seeds, a repeated-run count for every main-table cell, or uncertainty intervals for Table 4.^[1][7][13]

Evidence limits and independent commentary

The paper names several limits. Its signal and regularizer decomposition assumes components that may interact during gradient accumulation. All experiments are single-agent. A policy could learn to inflate the filtering statistic, environment randomness can make reward variance misleading, and aggressive selection may narrow exploration. The keep mass must be tuned for the task.^[1]

Nebius Science selected RAGEN-2 for its July 2026 ICML Papers That Matter roundup and summarized the mutual-information diagnosis and reward-variance filter. That article is independent conference commentary, not a re-execution of the experiment suite.^[14]

Hou's review likewise discusses the paper rather than reproducing it. It endorses the input-dependence diagnostic but questions whether the SNR account explains why weak task gradients produce fluent templates rather than ordinary failure. It also notes that high retrieval accuracy can accompany input-specific errors and that skipping weak-signal prompts does not solve an inadequate reward.^[15]

The independent sources located for this article did not report a full reproduction of the paper's main result matrix. The available code and experiment guides make inspection and rerunning possible, but the reported numbers remain author results unless another group executes and documents a matched protocol.^{[7][10][14][15]}

References

1. ^Wang, Zihan, et al. "RAGEN-2: Reasoning Collapse in Agentic RL." arXiv:2604.06268. Submitted April 7, 2026. arxiv.org/...2604.06268
2. ^International Conference on Machine Learning. "RAGEN-2: Reasoning Collapse in Agentic RL," ICML 2026 oral record. July 2026. icml.cc/...71062
3. ^International Conference on Machine Learning. "Understanding Reasoning Collapse in LLM Agent Reinforcement Learning," ICML 2026 poster record. July 2026. icml.cc/...66821
4. ^Microsoft Research. "RAGEN-2: Reasoning Collapse in Agentic RL." 2026. microsoft.com/...-reasoning-collapse-in-agentic-rl
5. ^Wang, Zihan, et al. "RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning." arXiv:2504.20073. Submitted April 24, 2025. arxiv.org/...2504.20073
6. ^MLL Lab. "RAGEN-2: Reasoning Collapse in Agentic RL," project page. Accessed August 21, 2026. ragen-ai.github.io
7. ^MLL Lab. "RAGEN," repository snapshot at commit 20daedc47558e000f7de912b060646bf2e8026bd. April 14, 2026. github.com/...edc47558e000f7de912b060646bf2e8026bd
8. ^MLL Lab. "Rollout Filtering Guide," RAGEN repository snapshot. Accessed August 21, 2026. github.com/...guide_rollout_filtering.md
9. ^MLL Lab. "collapse_metrics.py," RAGEN repository snapshot. Accessed August 21, 2026. github.com/...collapse_metrics.py
10. ^MLL Lab. "Main Table Runs," RAGEN repository snapshot. Accessed August 21, 2026. github.com/...experiment_main_table.md
11. ^MLL Lab. "setup_ragen.sh," RAGEN repository snapshot. Accessed August 21, 2026. github.com/...setup_ragen.sh
12. ^RAGEN Team. "MIT License," RAGEN repository snapshot. 2025. github.com/...LICENSE
13. ^MLL Lab. "RAGEN Releases." Accessed August 21, 2026. github.com/...releases
14. ^Sokolov, Arseniy, and Linda Petrini. "ICML 2026: Papers That Matter." Nebius Science. July 8, 2026. nebius.science/...icml-papers-that-matter
15. ^Hou, Qinglin. "Template Collapse in Agentic RL." May 23, 2026. qinglinh.com/...ragen2-template-collapse