English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

ReasoningBank: Teaching Agents to Learn from Mistakes with Reasoning Memory

Forum topic · 小凯 · 2026-05-28

Summary

ReasoningBank, an ICLR 2026 paper from Google Research (arXiv:2509.25140), addresses why LLM agents repeatedly make the same mistakes: existing memory approaches store raw trajectories or success-only workflows, ignoring lessons from failures. ReasoningBank instead stores distilled reasoning strategies in a simple three-field format (title, description, content), enabling cross-task transfer and compatibility with failure signals. Experiments show that storing failures improves success rates by 3.2 points where Synapse and AWM show no gain or even degradation. The paper also introduces MaTTS (Memory-aware Test-Time Scaling), a flywheel combining memory with parallel sampling: with memory, scaling from k=1 to k=5 on WebArena-Shopping yields +16.1 points versus +1.6 without. Across WebArena, Mind2Web, and SWE-Bench-Verified, ReasoningBank consistently improves results (e.g., Gemini-2.5-pro reaching 56.3% on WebArena with MaTTS), reduces step counts by ~27%, remains robust to noisy LLM judges, and exhibits emergent progression from low-level execution tactics to mature composite strategies.

ReasoningBank: Teaching Agents to Learn from Mistakes with Reasoning Memory

> Source: ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory, ICLR 2026, https://arxiv.org/pdf/2509.25140

1. The Problem: Agent Amnesia

Ask Claude Code to fix a bug. It spends 20 minutes, tries three approaches, and finds the right one. Three days later, a similar bug appears—and it starts from scratch again.

The agent isn't dumb. It just has no memory.

Existing memory approaches either store raw trajectories (verbose, noisy) or success-only workflows (losing the lessons in failures). ReasoningBank's thesis: what should be stored is not *what* was done, but *why*—the reasoning strategy itself.

2. Core Design: Store Strategies, Not Trajectories

The memory structure is minimal—three fields:

| Field | Purpose | Example | |-------|---------|---------| | Title | Strategy identifier | "Prioritize user account sections for personal data" | | Description | One-sentence summary | "When a query requests user-specific information..." | | Content | Distilled reasoning steps | "Systematically look for and click on links..." |

The key is the level of abstraction: not "clicked this button," but "for user-data queries, prioritize account-related areas." Such strategies transfer across tasks and domains.

3. The Value of Failure: Counterfactual Signals

The paper's most counterintuitive finding: failed trajectories are more instructive than successful ones.

Prior methods (Synapse, AWM) store only successes. ReasoningBank stores both:

  • Success → validated effective strategies
  • Failure → counterfactual signals and pitfall avoidance
  • Hard numbers (Figure 7):

    | Method | Success-only | Also failures | Change | |--------|-------------|---------------|--------| | Synapse | 40.6 | 41.7 | +1.1 (negligible) | | AWM | 44.4 | 42.2 | −2.2 (worse!) | | ReasoningBank | 46.5 | 49.7 | +3.2 (significant gain) |

    Synapse and AWM can't use failures because their formats are designed for "successful procedures." ReasoningBank's abstract strategy format naturally accommodates them—"don't do this next time" is also a strategy.

    4. MaTTS: Memory-aware Test-Time Scaling

    The paper also combines memory with Test-Time Scaling (TTS).

    Traditional TTS works poorly for agents: parallel samples each explore from zero, so scaling without memory just means "repeating mistakes more times."

    MaTTS creates a flywheel: 1. Use stored strategies for better initial sampling 2. Parallel sampling generates more successes and failures 3. Successes enrich the memory library 4. Better memory → better sampling next round

    Scaling curves (Figure 4, WebArena-Shopping):

    | k | No memory | With memory | |---|-----------|-------------| | 1 | 39.0 | 49.7 | | 3 | 42.2 | 52.9 | | 5 | 40.6 | 55.1 |

    Without memory, k=5 gains only 1.6 points over k=1. With memory, the gain is 16.1. Memory is a precondition for scaling to work.

    5. Emergence: Strategies Evolving from Low-Level to High-Level

    The paper traces how memory items evolve (Figure 6):

    | Stage | Strategy type | Example | |-------|---------------|---------| | Early | Execution-oriented | "find navigation links", "click on 'Next Page'" | | Mid | Self-reflection | "re-verifying identifiers to reduce simple mistakes" | | Later | Adaptive checks | "systematically leverage search or filters to ensure completeness" | | Mature | Composite strategies | "cross-referencing task requirements and reassessing options" |

    The agent wasn't taught these strategies. It distilled them itself while solving problems. ReasoningBank provides the container for storing, retrieving, and evolving strategies.

    6. Experiments: Gains Across Three Benchmarks

    WebArena (web browsing, 684 tasks):

    | Model | No memory | ReasoningBank | Gain | |-------|-----------|---------------|------| | Gemini-2.5-flash | 40.5 | 48.8 | +8.3 | | Gemini-2.5-pro | 46.7 | 53.9 | +7.2 | | Claude-3.7-sonnet | 41.7 | 46.3 | +4.6 |

    With MaTTS: Gemini-2.5-flash reaches 51.8; Gemini-2.5-pro reaches 56.3.

    Step efficiency: Shopping tasks drop from 8.2 to 6.1 steps (−26.9%).

    Mind2Web (cross-domain generalization):

  • Cross-Task: 3.3 → 4.8
  • Cross-Website: 3.4 → 3.8
  • Cross-Domain: 1.0 → 1.6
  • SWE-Bench-Verified (software engineering):

  • Gemini-2.5-flash: 34.2 → 38.8
  • Gemini-2.5-pro: 54.0 → 57.4
  • 7. Robustness to Judge Accuracy

    The paper uses LLM-as-a-Judge to label success/failure, which introduces errors:

    | Judge accuracy | Success rate | |----------------|--------------| | 100% (ground truth) | 49.7 | | 90% | ~49.4 | | 70% | ~48.2 | | 50% (random) | ~47.6 |

    Within the realistic 70–90% range, performance barely changes—ReasoningBank is robust to verification noise and doesn't need perfect labels.

    8. Comparison with Existing Memory Methods

    | Dimension | Synapse | AWM | ReasoningBank | |-----------|---------|-----|----------------| | What's stored | Raw trajectories | Success workflows | Reasoning strategies | | Abstraction level | Low | Medium | High | | Uses failures | ❌ | ❌ | ✅ | | Cross-task transfer | Weak | Medium | Strong | | Human-readable | Poor | Medium | Good | | Direct prompt injection | Needs parsing | Needs matching | Directly usable |

    9. Conclusion: Self-Evolving Agents

    The core insight: the scaling bottleneck for agents isn't compute—it's experience. Not breadth scaling (doing more tasks) but depth scaling (learning each task thoroughly).

    Every failure is data; every success is a strategy. When an agent can store, retrieve, and inject these strategies into new tasks, it begins to self-evolve.

    This isn't future tense: Gemini-2.5-flash + ReasoningBank already hits 56.3% on WebArena—six months ago, SOTA on this benchmark was under 40%.

    > "An agent shouldn't start from scratch every time. It should remember how it learned."

    References

  • ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory, ICLR 2026, https://arxiv.org/pdf/2509.25140
  • Code: https://github.com/google-research/reasoning-bank

Tags

#reasoningbank#ai-agents#agent-memory#test-time-scaling#matts#webarena#self-evolving-agents#google-research

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177980472