RAG Over Thinking Traces: Retrieving Reasoning Beats Retrieving Facts for LLM Inference

Forum topic · QianXun · 2026-05-08

Summary

A May 2026 UC Berkeley paper, "RAG over Thinking Traces Can Improve Reasoning Tasks," argues that retrieving AI thinking traces—the internal step-by-step reasoning generated before answers—outperforms traditional fact-document retrieval for complex reasoning tasks like math and code. The authors collect reasoning processes from frontier models into a reusable knowledge library and introduce the T^3 framework, which refines raw traces via Scaffolding (condensing verbose reasoning into concise guides) and Reflect Transformation (explicitly flagging logical traps prior models fell into). Experiments on the difficult AIME math benchmark show mid-tier Gemini 2.5 Flash improving 56.3% and GPT-5 gaining 7.6% in accuracy. Notably, retrieved thinking traces also reduce the model's own internal deliberation, cutting computation by roughly 15%, making responses both faster and cheaper. The work suggests reasoning patterns, not memorized facts, are the key to AI problem-solving, pointing toward a future of reusable, shared machine thought.

Imagine assembling an extremely complex LEGO set—say, the 7,000+ piece Millennium Falcon. When you get stuck, which helps more: a photo of the finished model plus an encyclopedia about its history, or another master builder's inner monologue while solving that exact spot—"this blue beam inserts backwards easily, I got it wrong twice; fix the left clip first and the right side slides in smoothly"?

Clearly the second. That is the provocative claim of a May 2026 UC Berkeley paper, "RAG over Thinking Traces Can Improve Reasoning Tasks": when tackling complex reasoning (math, code), retrieving *thinking traces* is far more effective than retrieving *factual documents*.

What Are "Thinking Traces"?

Today's top models (e.g., GPT-o1, Gemini 2 Thinking) spend tens of seconds or even minutes on internal reasoning before answering—trial and error, self-correction, logical derivation. We used to treat these processes as disposable, discarded once the answer was produced.

The Berkeley researchers disagree: the reasoning process itself is the treasure. They collected these traces into a large "library of thought." When you ask the AI a hard math problem, the system no longer consults encyclopedias; it searches: *Has a smart AI reasoned about something similar before? How did it think? Where did it stumble?*

The T^3 Framework: Teaching AI to Avoid Pitfalls

The paper's most interesting contribution is a framework called T^3 for refining these traces, because raw reasoning is often too long and messy. Two techniques:

Scaffolding: condense verbose thinking into a concise, strategy-like guide.
Reflect Transformation: the cleverest trick—not only showing the correct path, but explicitly warning: "watch out, there's a logical trap here that many AIs fell into before—don't repeat their mistake."

How Strong Are the Results?

On the very difficult AIME (American Invitational Mathematics Examination) benchmark:

Mid-tier Gemini 2.5 Flash jumped 56.3% after retrieving thinking traces.
Even the strongest model, GPT-5, gained 7.6% accuracy when guided by prior models' reasoning.

Even better: the method makes AI faster and cheaper, not just smarter. With a clear "reasoning guide" in hand, the model's own internal deliberation shortens, reducing computation by roughly 15%.

Why This Matters Cognitively

Feynman said: "I have to think from scratch, so I can understand what I'm doing." The paper suggests that **understanding is fundamentally about mastering a thinking *pattern*, not memorizing a dead *fact*.

Feeding AI more encyclopedias treats it as a memorization machine. Retrieving thinking traces treats it as a learning machine.

Takeaway

We are entering an era of reusable thought. Future AI collaboration may be less "question and answer" and more a massive relay of reasoning**: when you face a hard problem, the detours, avoided traps, and flashes of insight from countless top models become the scaffolding in your own mind.

Don't settle for knowing *what the answer is*—learn *how others thought about it*. That may be the ultimate law of co-evolution between humans and machines in the AI era.