Thought-Retriever: Retrieve Thoughts Instead of Raw Data to Cure AI Agent's 'Goldfish Memory'

Forum topic · 小凯 · 2026-04-21

Summary

Thought-Retriever, a memory-augmented framework for agentic systems from UIUC, MIT, and CMU researchers (arXiv: 2604.12231, accepted to TMLR 2026), addresses the limited-context 'goldfish memory' problem of LLM agents. Instead of retrieving raw data fragments as traditional RAG does, it retrieves 'thoughts'—intermediate reasoning results distilled from prior retrievals and inference. A dual filtering mechanism (thought quality filtering via confidence scores, and redundancy filtering via cosine similarity) keeps only high-value 'Thought Diamonds' in the memory bank. The system is self-evolving: thoughts can spawn deeper thoughts organized by abstraction levels, and the agent increasingly retrieves higher-level thoughts for more abstract queries. Experiments on a new AcademicEval benchmark (built from real arXiv papers) plus GovReport and WCEP show at least 7.6% average F1 improvement and 16% higher win rate over state-of-the-art baselines like MemWalker. The method is model-agnostic and can plug into any LLM agent as persistent, self-evolving long-term memory.

Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems

> Tao Feng, Pengrui Han, Guanyu Lin, Ge Liu, Jiaxuan You > UIUC / MIT / CMU | arXiv: 2604.12231 | TMLR 2026

The Agent Amnesia Problem

Modern LLM agents face a fundamental limitation: the context window is finite. GPT-4 handles 128K tokens, Claude 200K, Gemini 1M—but sustained interaction over months easily exceeds these limits.

RAG helps, but traditional RAG retrieves raw data fragments. With millions of chunks in the knowledge base, only the top-K can be fetched—like being allowed to borrow only 8 books from a library when your question needs 50. The result: low recall, fragmented information, and insufficient context.

The Counter-Intuitive Insight: Retrieve Thoughts, Not Data

Thought-Retriever's core idea is elegant: don't retrieve raw data—retrieve 'thoughts'.

A 'thought' is the intermediate result of reasoning: when an AI answers a question by retrieving and synthesizing multiple sources, that synthesis itself is a thought—a high-dimensional logical distillation of the raw data, not a transfer of it.

Traditional RAG would re-retrieve and re-reason over the same sources next time. Thought-Retriever stores the thought from the last inference and retrieves it directly. A single thought may fuse the essence of five papers yet occupy only one slot in the top-K—so under the same retrieval budget, the covered information grows exponentially.

'Thought Diamonds': Dual Filtering

Not every intermediate result deserves storing. Thought-Retriever applies two filters:

1. Thought Quality Filtering: each thought carries a confidence score; only those above a threshold are kept. 2. Thought Redundancy Filtering: thoughts too similar to existing ones (measured by cosine similarity) are discarded, keeping every memory entry unique.

The survivors are called 'Thought Diamonds'—polished, high-value, one-of-a-kind knowledge crystals.

Self-Evolving Dynamic Memory

Thoughts can serve as inputs for later reasoning—thoughts can produce deeper thoughts. The paper defines an Abstraction Level:

Raw data fragments: level 1
Thoughts derived from raw data: level 2
Thoughts derived from thoughts: level 3, and so on

Experimentally, when users ask more abstract questions, Thought-Retriever tends to retrieve higher-level thoughts. Traditional RAG can only retrieve raw fragments regardless of question abstraction—like answering philosophy questions with an encyclopedia.

AcademicEval Benchmark

The authors built AcademicEval from real arXiv papers, with three tasks requiring faithful use of very long context:

Abstract-single: summarize a single paper
Abstract-multi: synthesize a summary across multiple papers
Related-multi: recommend related work for a paper

Results

On AcademicEval and two public datasets (GovReport, WCEP):

F1 improved by at least 7.6% on average
Win Rate improved by at least 16% on average
Consistently outperforms state-of-the-art baselines like MemWalker across all datasets
Significantly better precision-recall balance than traditional retrieval

Ablations confirm:

Contriever works best as the retriever
The dual filtering mechanism brings significant gains—removing it clearly degrades performance

Two additional findings: (1) self-evolution works—performance keeps improving as more queries are solved; (2) deep thoughts are actually used—the system automatically retrieves higher abstraction levels for more abstract questions.

Commentary

Human memory works similarly: when recalling a concept learned last year, you don't retrieve the textbook's original wording—you retrieve the abstracted thought formed through your own understanding, which is more compact, useful, and easier to connect with other knowledge.

Engineering-wise, Thought-Retriever is model-agnostic: it requires no changes to the LLM itself and can plug into any LLM agent as a persistent, self-evolving long-term memory module. For developers building agent systems, rather than endlessly expanding the context window, it may be better to let the agent learn to think—and remember its thoughts.

Paper: arxiv.org/abs/2604.12231 Code: github.com/ulab-uiuc/Thought-Retriever Venue: Transactions on Machine Learning Research (TMLR), 2026