ReasoningBank

ReasoningBank is a memory framework for Large language model (LLM) AI agents that distills generalizable reasoning strategies from successful and failed experiences. It was proposed by researchers affiliated with Google Cloud AI Research, the University of Illinois Urbana-Champaign, Yale University, and Google Cloud AI. The paper reports a 34.2% relative success-rate gain for ReasoningBank with MaTTS in the sequential k=5 comparison, 54.5% versus 40.6% for the no-memory scaling baseline. For ReasoningBank without MaTTS, the largest overall WebArena gain was 8.3 percentage points, from 40.5% to 48.8%, or about 20.5% relative. The reported overall interaction-step reduction reached about 16%.^[1]

The framework addresses a fundamental limitation in current LLM agents: their inability to learn from accumulated interaction history, forcing them to discard valuable insights and repeat past errors. As these agents are increasingly deployed in persistent, long-running roles, they naturally encounter continuous streams of tasks throughout their lifetime. However, they largely fail to learn from their accumulated experience across tasks, approaching each task in isolation and lacking self-evolving capabilities that make the agent system more capable over time.^[1]

Overview

ReasoningBank introduces two key innovations that distinguish it from prior memory approaches. First, it structures memory as human-interpretable knowledge units containing titles, descriptions, and distilled reasoning content, abstracting away low-level execution details while preserving transferable patterns. Unlike traditional memory systems that store raw trajectories or only successful routines, ReasoningBank extracts high-level reasoning strategies and decision rationales from both positive and negative experiences.^[1]

Second, it integrates with memory-aware test-time scaling (MaTTS), which accelerates learning by generating abundant diverse experiences that provide rich contrastive signals for synthesizing higher-quality memory. This establishes memory-driven experience scaling as a new scaling dimension for agents, complementing traditional parameter scaling and data scaling approaches.^[1]

Building on ReAct^[2] agent architecture, ReasoningBank operates as a closed-loop system: agents retrieve relevant memories before tasks, execute with memory guidance, then extract new learnings to enrich the memory bank. This closed-loop process of retrieval, execution, and consolidation allows the agent to continuously evolve and improve its strategic capabilities, enabling agents to self-evolve with emergent behaviors arising naturally over time, progressing from execution-oriented strategies to adaptive self-reflections and compositional reasoning without parameter updates.^[1]

Background

With the growing adoption of large language model agents in persistent real-world roles, they naturally encounter continuous streams of tasks. A key limitation, however, is their failure to learn from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors. By approaching each task in isolation, they are doomed to repeat past mistakes, discard valuable insights from related problems, and lack self-evolving capabilities that make the agent system more capable over time.^[1]

Recent efforts on agent memory have primarily focused on storing past interactions for reuse. While useful, these approaches are often limited to leveraging raw trajectories or common, successful routines (i.e., workflows, procedures). These approaches suffer from two fundamental drawbacks. First, they lack the ability to distill higher-level, transferable reasoning patterns. Second, by over-emphasizing successful experiences, they leave the valuable lessons from an agent's own failures largely underexplored. Consequently, existing memory designs often remain limited to passive record-keeping rather than providing actionable, generalizable guidance for future decisions.^[1]

Methodology

Core framework

ReasoningBank implements a three-stage closed-loop process integrating memory systems for AI agents with agent execution:

Memory Retrieval: Before task execution, the agent queries ReasoningBank using embedding-based similarity search (Gemini-Embedding-001 with cosine distance) to retrieve the top-k relevant memory items (default k=1). Retrieved items are injected into the agent's system instruction to guide decision-making, ensuring that the decision-making is grounded with useful past experiences.^[1]

Memory Construction: After task completion, the system obtains proxy signals for the correctness of completed trajectories using an LLM-as-a-judge (no ground-truth labels required). Different extraction strategies apply for successes versus failures: successful experiences contribute validated strategies, while failed experiences supply counterfactual signals and pitfalls that help sharpen guardrails. The system extracts up to three memory items per trajectory using the same backbone LLM as the agent at temperature 1.0 for extraction.^[1]

Memory Consolidation: New memory items are incorporated into ReasoningBank with a simple addition operation, maintaining an evolving repository of memory items. Items are appended to ReasoningBank in JSON format with pre-computed embeddings stored separately for efficient retrieval. The consolidation deliberately uses simple addition operations to isolate the contribution of memory content quality rather than architectural complexity.^[1]

Together, these steps form a closed-loop process: the agent leverages past experiences, constructs new memory from current tasks, and continually updates its memory, enabling sustained evolvement in test-time learning scenarios.

Memory schema

Each memory item in ReasoningBank is designed and induced from past experiences as structured knowledge units that abstract away low-level execution details while preserving transferrable reasoning patterns and strategies. Items are stored as structured triples containing three components:^[1]

• Title: A concise identifier summarizing the core strategy or reasoning pattern

• Description: A brief one-sentence summary of the memory item that provides context or purpose

• Content: Distilled reasoning steps, decision rationales, or operational insights extracted from past experiences

This schema design ensures memory items remain both human-interpretable and machine-usable while abstracting away task-specific details. The content preserves transferable reasoning patterns that generalize across similar scenarios, capturing both effective strategies from successes and crucial preventative lessons from failures. Together, memory items extracted are both human-interpretable and machine-usable, facilitating efficient usage and integration with agents.^[1]

Integration with agents

An agent equipped with ReasoningBank can draw upon a curated pool of transferable strategies to guide decision-making. This enables the agent to recall effective insights, avoid previously observed pitfalls, and adapt more robustly to unseen queries.^[1]

ReasoningBank demonstrates compatibility across multiple LLM backbones including Gemini-2.5-Flash, Gemini-2.5-Pro, and Claude-3.7-Sonnet, all accessed via Vertex AI API. The agent policy is conditioned on the memory module, which provides relevant insights to inform actions. Agents follow the ReAct style implementation with interleaved reasoning and acting, generating trajectories of observations and actions for each given task.^[1]

The framework distinguishes itself from related approaches:

• Synapse^[3]: Stores complete raw trajectories with state abstraction, organizing past trajectories as in-context memory

• Agent Workflow Memory (AWM)^[4]: Abstracts common patterns from successful trajectories into reusable workflows and procedural routines

• ReasoningBank: Distills strategy-level reasoning from both successes and failures, enabling learning from counterfactual signals and capturing high-level, transferable reasoning patterns beyond procedural workflows

MaTTS: memory-aware test-time scaling

Memory-aware test-time scaling (MaTTS) creates synergy between memory quality and scaled exploration by deliberately learning from abundant trajectories generated during test-time compute allocation. Instead of scaling experience through breadth by adding more tasks, MaTTS focuses on scaling experience through depth by tackling each single task with more exploration. Unlike vanilla test-time scaling that independently converts trajectories to memory, MaTTS uses contrastive signals across multiple attempts to curate more effective memory.^[1]

MaTTS deliberately learns from the abundant successful and failure trajectories generated during scaling for more effective memory curation. The system introduces two complementary instantiations: parallel scaling and sequential scaling.^[1]

Parallel scaling

Generates multiple trajectories (k) for the same query simultaneously under guidance of retrieved memory items. By comparing and contrasting (self-contrast) across different trajectories, the agent can identify consistent reasoning patterns while filtering out spurious solutions. This process enables more reliable memory curation from multiple trials of a single query that promotes diverse exploration.^[1]

Sequential scaling

Iteratively refines reasoning within a single trajectory after initial completion, following the principle of self-refinement. During this process, the intermediate notes generated in self-refinement are also used as valuable signals for memory, since they capture reasoning attempts, corrections, and insights that may not appear in the final solution.^[1]

The scaling factor k denotes the number of trajectories for parallel scaling and refinement steps for sequential scaling. Equipped with ReasoningBank, both parallel and sequential strategies become memory-aware, ensuring that the additional computation allocated at test time translates into more transferable and higher-quality memory for future tasks.

Synergistic effects

MaTTS establishes a powerful synergy between memory and test-time scaling: high-quality memory from ReasoningBank steers scaled exploration toward more promising paths, while the rich experiences generated during scaling forge even stronger memories. This positive feedback loop positions memory-driven experience scaling as a new scaling dimension for agents.^[1]

At k=5 parallel scaling on WebArena-Shopping with Gemini-2.5-Flash, Best-of-5 rose from 39.0% without memory and scaling to 42.2% without memory, from 40.6% to 44.4% with Synapse, from 44.4% to 47.6% with AWM, and from 49.7% to 55.1% with ReasoningBank. The largest absolute gain occurred with ReasoningBank.^[1]

After the scaled rollouts were used for memory curation, Pass@1 remained 39.0% without memory, rose from 40.6% to 41.2% with Synapse, from 44.4% to 45.5% with AWM, and from 49.7% to 53.0% with ReasoningBank. All three memory methods improved in this v2 analysis, but the gain was largest for ReasoningBank.^[1]

Experiments

Benchmarks

Experiments evaluated ReasoningBank across three diverse, challenging benchmarks for web browsing and software engineering tasks:

WebArena^[5]: The study used 684 web-navigation tasks after its stated exclusions: shopping (187), administration (182), GitLab (180), Reddit (106), and multi-domain tasks (29). WebArena provides functional, self-hosted websites for realistic tasks.^[5] The ReasoningBank agents ran in the BrowserGym^[6] environment with text-based accessibility-tree observations and a maximum of 30 steps per query.^[1]

Mind2Web^[7]: The study evaluated 1,341 test instances: Cross-Task (252), Cross-Website (177), and Cross-Domain (912). The full Mind2Web dataset contains more than 2,000 tasks from 137 websites across 31 domains.^[7] The reported metrics were element accuracy (EA), action F1 (AF1), step success rate (SSR), and task-level success rate (SR).^[1]

SWE-bench Verified^[8][9]: The original SWE-bench evaluates issue resolution using real GitHub repositories and issue descriptions.^[8] OpenAI and the SWE-bench authors produced the 500-instance Verified subset through human review.^[9] ReasoningBank evaluated agents in a bash-only Mini-SWE-Agent environment.^[1]

Evaluation setup

All experiments used ReAct agents with decoding temperature of 0.7 for model generations on web browsing tasks and temperature of 0.0 for deterministic LLM-as-a-judge evaluation. For memory extraction, temperature was set to 1.0. Baselines included:

• No Memory: The backbone LLM agent without any memory module, serving as a reference point

• Synapse: A representative work that organizes past trajectories as in-context memory

• AWM: Abstracts common patterns from trajectories into reusable workflows

Together, these baselines span a progression from agents without memory, to those that directly reuse past trajectories, and finally to methods that distill higher-level structures, providing a comprehensive comparison for evaluating ReasoningBank. Evaluation metrics measured both effectiveness (success rates, resolve rates) and efficiency (interaction steps required to complete tasks).^[1]

Results

Performance improvements across benchmarks

ReasoningBank consistently outperformed all baselines across three model backbones (Gemini-2.5-Flash, Gemini-2.5-Pro, Claude-3.7-Sonnet) and all benchmarks:

Scroll sideways for more →

Specifically, ReasoningBank improved the overall success rate on WebArena by +8.3, +7.2, and +4.6 percentage points with three different backbone LLMs compared to memory-free agents, representing relative improvements of approximately 20.5%, 15.4%, and 11.0% respectively.^[1]

Scroll sideways for more →

A similar pattern holds on Mind2Web, where ReasoningBank delivers clear gains across cross-task, cross-website, and cross-domain settings, underscoring both the consistency and scalability of its benefits across datasets and model sizes. The gains are especially pronounced in the cross-domain setting, which requires the highest level of generalization.^[1]

Scroll sideways for more →

Results on SWE-Bench-Verified further confirm ReasoningBank's robustness on repository-level software engineering tasks.^[1]

The WebArena Multi subset requires transfer across multiple websites. Averaged across the three backbones, the paper reports that ReasoningBank gained 4.6 percentage points over the strongest baseline. For Gemini-2.5-Flash specifically, ReasoningBank achieved 13.8% versus 10.3% for the strongest baseline, while AWM achieved 3.4%.^[1]

Enhanced generalization

ReasoningBank enhances generalization with better transferrable memory across tasks, particularly in challenging generalization settings. The memory curated by ReasoningBank is more robust and transferable, enabling agents to generalize effectively across diverse scenarios. These results demonstrate that unlike baselines, ReasoningBank memory items are not tied to specific websites or queries but capture generalizable reasoning patterns.^[1]

Efficiency gains

In addition to higher success rates, ReasoningBank achieves superior efficiency by leveraging past experiences as memory. It reduces the number of interaction steps needed to complete tasks, as shown in the Step metrics across benchmarks. On WebArena, across almost all subsets and backbones, ReasoningBank lowers the average step count by up to 1.4 compared with "No Memory", and 1.6 compared with other memory baselines.^[1]

Scroll sideways for more →

The reduction is particularly pronounced on successful cases, reaching up to 2.1 fewer steps (a 30.9% relative reduction on Shopping domain) compared to failed ones. This indicates that ReasoningBank primarily helps the agent reach solutions with fewer interactions by strengthening its ability to follow effective reasoning paths rather than simply truncating failed trajectories. This highlights the role of memory in guiding purposeful decision-making and improving efficiency in practice.^[1]

Learning from failures

ReasoningBank uniquely transforms failures into constructive signals through its design that enables distillation of reasoning patterns from both successes and failures. An ablation study on WebArena-Shopping with Gemini-2.5-flash compared different memory designs under two settings: using only successful trajectories versus leveraging both successes and failures.^[1]

Scroll sideways for more →

In this experiment, Synapse improved only slightly when failed trajectories were added, while AWM declined. ReasoningBank gained 3.2 percentage points, consistent with its design goal of extracting constructive signals from both successful and failed trajectories.^[1]

MaTTS scaling effectiveness

Memory-aware test-time scaling amplified ReasoningBank's advantages substantially. Experiments on WebArena-Shopping with Gemini-2.5-flash demonstrated the effect of scaling factor k for both parallel and sequential scaling:

The paper reports endpoint and range comparisons for both scaling modes.^[1]

With ReasoningBank, parallel MaTTS rose from 49.7% at k=1 to 55.1% at k=5, while sequential MaTTS rose from 49.7% to 54.5%. The no-memory baseline was less consistent: parallel results ranged from 39.0% to 42.2%, and sequential results ranged from 37.4% to 40.6%.^[1]

At k=5, MaTTS also exceeded vanilla test-time scaling without aggregation: 55.1% versus 52.4% for parallel scaling, and 54.5% versus 51.9% for sequential scaling. Sequential refinement was stronger at small k, but parallel scaling was stronger at k=5.^[1]

Synergy between memory and scaling

The experimental results reveal a bidirectional synergy between memory quality and test-time scaling effectiveness:

Better memory enables stronger test-time scaling performance. At k=5, Best-of-5 reached 42.2% without memory, 44.4% with Synapse, 47.6% with AWM, and 55.1% with ReasoningBank. Relative to each method's no-scaling result, the gains were 3.2, 3.8, 3.2, and 5.4 percentage points, respectively.^[1]

Scaling yields better memory curation. Pass@1 after scaled memory curation was unchanged at 39.0% without memory, but rose by 0.6 points with Synapse, 1.1 points with AWM, and 3.3 points with ReasoningBank. The paper interprets the larger ReasoningBank gain as evidence of a bidirectional relationship between its memory design and scaled exploration.^[1]

Emergent behaviors

The strategies in ReasoningBank are not flat or monolithic, but instead evolve over time, exhibiting emergent behaviors that resemble the learning dynamics of reinforcement learning. A memory item in ReasoningBank gradually evolves during the test-time learning process, progressing through distinct stages:^[1]

1. Execution-oriented/procedural strategies: The agent follows straightforward action rules (for example finding navigation links, clicking specific elements)

2. Adaptive self-reflections: Re-verifying identifiers to reduce simple mistakes, checking element correctness before actions

3. Adaptive checks: Systematically leveraging available search or filter functions to ensure completeness before reporting results

4. Compositional strategies: Cross-referencing task requirements with current views and reassessing available options, handling misaligned data through multiple reasoning paths

This hierarchical evolution highlights how ReasoningBank enables agents to refine strategies from low-level actions to high-level reasoning during test-time learning without any parameter updates. The agent develops increasingly complex and effective reasoning strategies through the accumulation and synthesis of experiences in its memory bank.^[1]

Related work

Memory frameworks for agents

Memory has emerged as an essential module in modern agent systems to enhance their performance by utilizing past information. Existing memory systems organize and store information in various forms, including plain text, latent knowledge embeddings, and structured graphs. Beyond memory content, those methods usually involve retrieval mechanisms (for example semantic search) with memory management strategies (for example updating).^[1]

Prior work explored various memory mechanisms for LLM agents. Most efforts primarily emphasize personalization and long-context management. Trajectory-based approaches like Synapse^[3] store complete interaction histories with state abstraction, while workflow-based methods like AWM^[4] extract reusable task sub-routines and procedural workflows from successful trajectories only. Other frameworks employ latent embeddings, structured knowledge graphs, or reinforcement learning for memory management.^[1]

ReasoningBank differentiates by storing high-level strategies and reasoning hints rather than raw trajectories or procedural workflows. By abstracting experiences into reusable reasoning units, ReasoningBank enables agents to generalize not only from successful cases but also by learning from failures, thereby providing richer guidance for test-time learning. Different from previous works that emphasize reusing successful trajectories or procedural workflows, ReasoningBank stores strategy-level reasoning that captures transferable patterns and uniquely learns from both successes and failures. The authors present the framework as the first exploration of memory-aware test-time scaling for agents.^[1]

Test-time scaling for language models

Test-time scaling has demonstrated strong effectiveness and has become a widely adopted practice in end-to-end problem-solving such as coding and math reasoning, where methods including best-of-N, beam search, and leveraging verifiers are commonly employed. Recent work demonstrated that allocating additional compute at inference time can rival scaling model parameters for reasoning tasks.^[1]

However, its application to multi-turn interactive scenarios, particularly agentic tasks, remains underexplored. Existing works mainly adapt lessons learned from reasoning tasks and scale different dimensions of agentic systems, including the search space for each action, the number of agents in multi-agent systems, and the number of interactions with the environment. None of these efforts considers the role of agent memory in scaling, where an agent can learn from past experiences to guide future decisions.^[1]

The authors describe MaTTS as the first integration of memory with test-time scaling for agents. In their experiments, memory and scaling produced complementary gains, which they frame as memory-driven experience scaling.^[1]

Publication and code

The preprint was first submitted on September 29, 2025 and revised on March 16, 2026.^[10] The paper was accepted to ICLR 2026, and the authors released an official code repository for the WebArena and SWE-bench experiments.^[1]

Limitations and future directions

Current limitations

While ReasoningBank demonstrates strong empirical performance, several limitations have been identified:^[1]

• Focus on memory content: The study emphasizes memory content (integrating failure trajectories, constructing distilled reasoning cues) rather than comparing with other memory architectures such as episodic or hierarchical memory

• Simplicity in retrieval and consolidation: Simple embedding-based retrieval and straightforward consolidation were intentionally adopted to isolate the effect of content quality

• Dependence on LLM-as-a-judge: Success and failure signals are determined by an LLM-based evaluator, which may introduce noise when tasks are ambiguous

Future directions

Several promising directions for future research have been identified:^[1]

Compositional Memory: The current framework distills each experience into multiple memory items retrieved and reused independently. Future work could explore composition-aware retrieval and consolidation, enabling agents to combine complementary items or form reusable macros, yielding richer strategies and stronger generalization in long-horizon tasks.

Advanced Memory Architectures: The current system design is intentionally minimal; a natural next step is building a layered, product-level memory stack that integrates mature paradigms: episodic traces for per-task context, short-term "working" memory for within-session state, and long-term consolidated knowledge with decay/refresh policies. Additionally, retrieval could move beyond embedding-based similarities to reasoning-intensive controllers that decompose queries, plan multi-hop lookups across tiers, and condition selection on uncertainty, recency, and cost.

Enhanced Verifiers: Future work could incorporate stronger verifiers, human-in-the-loop feedback, or ensemble judgment to enhance the reliability of memory induction beyond the current LLM-as-a-judge approach.

See also

• Large language model

• AI agent

• ReAct

• Test-time scaling

• Agent Workflow Memory

• Synapse (AI framework)

• WebArena

• Mind2Web

• SWE-Bench

• Chain-of-thought prompting

• Papers

• Memory systems for AI agents

References

1. ^Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. "ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory." International Conference on Learning Representations, 2026.
2. ^Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations, 2023.
3. ^Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. "Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control." International Conference on Learning Representations, 2024.
4. ^Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. "Agent Workflow Memory." Proceedings of the 42nd International Conference on Machine Learning, 2025.
5. ^Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. "WebArena: A Realistic Web Environment for Building Autonomous Agents." International Conference on Learning Representations, 2024.
6. ^Thibault Le Sellier de Chezelles et al. "The BrowserGym Ecosystem for Web Agent Research." Transactions on Machine Learning Research, 2025.
7. ^Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. "Mind2Web: Towards a Generalist Agent for the Web." Advances in Neural Information Processing Systems 36, 2023.
8. ^Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" International Conference on Learning Representations, 2024.
9. ^OpenAI. "Introducing SWE-bench Verified." August 13, 2024; updated February 24, 2025.
10. ^Siru Ouyang et al. "ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory." arXiv:2509.25140, submitted September 29, 2025; revised March 16, 2026.