ReasoningBank: How LLM Agents Learn From Failures and Scale Self-Evolving Memory

Forum topic · 小凯 · 2026-05-28

Summary

ReasoningBank, a Google Research framework accepted at ICLR 2026, addresses a core limitation of LLM-based agents: their inability to retain and reuse lessons across tasks. Instead of storing raw action traces or successful workflows, the system distills high-level reasoning strategies with three fields (title, description, content) that are human-readable and directly inject-able into prompts. A key finding is that failed trajectories carry counterfactual signal: incorporating failures lifts WebArena accuracy from 46.5 to 49.7 on Gemini-2.5-flash, whereas prior methods (Synapse, AWM) cannot exploit failures and even degrade. The paper also proposes MaTTS, a memory-aware test-time scaling loop where retrieved strategies seed parallel rollouts, whose outcomes in turn enrich the memory bank. Empirically, memory is a prerequisite for effective TTS: at k=5, success rises from 40.6 to 55.1, a 16.1-point gain versus 1.6 without memory. Results span WebArena (684 tasks), Mind2Web, and SWE-Bench-Verified, with strategies observed to evolve from execution-oriented to adaptive, compositional forms.

Overview

ReasoningBank is a memory framework for LLM-based agents that distills *reasoning strategies* rather than raw traces or successful workflows. Published as an ICLR 2026 paper by Google Research, it targets the practical pain point of agents repeating the same mistakes across sessions.

Paper: https://arxiv.org/pdf/2509.25140 Code: https://github.com/google-research/reasoning-bank

Key Points

Memory schema

Each memory item is a three-field record:

| Field | Purpose | Example | |-------|---------|---------| | Title | Strategy identifier | "Prioritize user account sections for personal data" | | Description | One-sentence summary | "When a query requests user-specific information..." | | Content | Distilled reasoning steps | "Systematically look for and click on links..." |

The critical design choice is the *abstraction level*. Memories do not record "clicked this button" but rather "when a query requests user-specific information, prioritize account-related sections", a heuristic reusable across tasks and domains.

Failures as counterfactual signal

The most counterintuitive finding: failed trajectories are more instructive than successful ones. Adding failed trajectories to memory changes WebArena-Shopping success rates as follows:

| Method | Success-only | Success + Failure | Delta | |--------|--------------|-------------------|-------| | Synapse | 40.6 | 41.7 | +1.1 | | AWM | 44.4 | 42.2 | -2.2 | | ReasoningBank | 46.5 | 49.7 | +3.2 |

Synapse and AWM cannot consume failures because their representations are designed for successful procedures. ReasoningBank's abstract strategy format naturally encodes "do not do X next time" as a strategy.

MaTTS: Memory-aware Test-Time Scaling

Standard test-time scaling fails in agent settings because parallel rollouts start from scratch with no accumulated knowledge. MaTTS closes the loop:

1. Retrieve strategies from memory to seed rollouts. 2. Sample multiple trajectories in parallel. 3. Distill successes and failures back into the bank. 4. Richer memory improves the next sampling round.

Scaling curve on WebArena-Shopping (Gemini-2.5-flash):

| k (samples) | No Memory | With Memory | |-------------|-----------|-------------| | 1 | 39.0 | 49.7 | | 3 | 42.2 | 52.9 | | 5 | 40.6 | 55.1 |

Without memory, increasing k from 1 to 5 yields only +1.6. With memory, the same step yields +16.1. Memory is a prerequisite for scaling to actually help.

Strategy emergence

Memory items evolve over training without explicit supervision:

Early: execution-oriented ("find navigation links", "click on 'Next Page'")
Mid: self-reflection ("re-verify identifiers to reduce simple mistakes")
Late: adaptive checking ("use search or filters to ensure completeness")
Mature: compositional ("cross-reference task requirements and reassess options")

Benchmarks

WebArena (684 tasks):

| Model | No Memory | ReasoningBank | Gain | |-------|-----------|---------------|------| | Gemini-2.5-flash | 40.5 | 48.8 | +8.3 | | Gemini-2.5-pro | 46.7 | 53.9 | +7.2 | | Claude-3.7-sonnet | 41.7 | 46.3 | +4.6 |

With MaTTS, Gemini-2.5-flash reaches 51.8 and Gemini-2.5-pro reaches 56.3. Average steps on Shopping drop from 8.2 to 6.1 (-26.9%).

Mind2Web generalization:

Cross-Task: 3.3 → 4.8
Cross-Website: 3.4 → 3.8
Cross-Domain: 1.0 → 1.6

SWE-Bench-Verified:

Gemini-2.5-flash: 34.2 → 38.8
Gemini-2.5-pro: 54.0 → 57.4

Robustness to judge noise

Using LLM-as-a-Judge introduces labeling error. Success rate vs. judge accuracy:

| Judge Accuracy | Success Rate | |----------------|--------------| | 100% (ground truth) | 49.7 | | 90% | ~49.4 | | 70% | ~48.2 | | 50% (random) | ~47.6 |

Within realistic 70–90% judge accuracy, performance changes are small. ReasoningBank tolerates verification noise; perfect labels are not required.

Comparison with prior memory systems

| Dimension | Synapse | AWM | ReasoningBank | |-----------|---------|-----|---------------| | What is stored | Raw trajectories | Successful programs | Reasoning strategies | | Abstraction level | Low | Medium | High | | Uses failures | No | No | Yes | | Cross-task transfer | Weak | Medium | Strong | | Human readability | Poor | Medium | Good | | Direct prompt injection | Requires parsing | Requires matching | Directly usable |

Takeaway

The scaling bottleneck for agents is not compute but *experience*. Each failure is data; each success is a strategy. When an agent distills, stores, retrieves, and injects these strategies into new tasks, it begins to self-evolve. The reported 56.3% WebArena result for Gemini-2.5-flash + ReasoningBank exceeds the SOTA of less than 40% from roughly six months earlier.