RAG Over Thinking Traces: A New Path to Better LLM Reasoning

Forum topic · QianXun · 2026-05-08

Summary

A May 2026 paper from UC Berkeley, "RAG over Thinking Traces Can Improve Reasoning Tasks," argues that retrieving AI thinking traces is far more effective for complex reasoning than retrieving factual documents. Instead of feeding models encyclopedic knowledge, the approach builds a library of reasoning processes generated by frontier models, letting systems find how similar problems were previously solved and which logical traps were encountered. The paper introduces the T^3 framework, which refines raw thinking traces via Scaffolding (condensing verbose reasoning into concise guides) and Reflect Transformation (explicitly warning about common mistakes). Experiments show striking gains: on the difficult AIME math benchmark, retrieval of thinking traces boosted Gemini 2.5 Flash by 56.3% and even improved GPT-5 accuracy by 7.6%, while cutting the model's own reasoning time and computation by roughly 15%. The finding suggests understanding rests on mastering thinking patterns rather than memorizing facts, pointing toward a future of reusable, transferable reasoning traces across AI systems.

Imagine assembling an extremely complex LEGO set, like the 7,000+ piece Millennium Falcon.

If you get stuck, there are two ways to help:

1. The first way (traditional RAG): I hand you a photo of the fully assembled model, or even dump the entire Millennium Falcon encyclopedia on you — its history, engine principles, and the name of every compartment. 2. The second way (the new RAG from the paper): I hand you a LEGO master's "inner monologue" while building that section. He tells you: "Hey, this blue beam is easy to insert backwards — I failed twice; if you lock the left clip first, the right side goes in much more smoothly."

Which helps you more with the problem at hand? Clearly the second.

That's the disruptive claim of a May 2026 paper from UC Berkeley — 《RAG over Thinking Traces Can Improve Reasoning Tasks》: when it comes to complex reasoning tasks like math and code, retrieving "thinking traces" is far more effective than retrieving "factual documents."

What Are "Thinking Traces"?

Today's top models (like GPT-o1, Gemini 2 Thinking) spend tens of seconds or even minutes "thinking internally" before answering. They trial-and-error, self-correct, and reason step by step.

We used to treat these thought processes as disposable — once the answer came out, they were tossed into the digital trash bin.

But the Berkeley researchers say: No — the thinking process itself is the real treasure!

They collected these thinking processes and built a massive "library of thought." When you ask the AI a hard math problem, the system no longer searches an encyclopedia; instead it searches this library: "Has a smart AI thought about a similar problem before? How did it think? Where did it stumble?"

Teaching AI to Avoid Pitfalls: The $\mathcal{T}^3$ Framework

The most elegant part of the paper is that they don't just stuff raw thinking traces into the AI — they built the $\mathcal{T}^3$ framework to refine them.

They found raw thinking traces are often too long and messy, so they optimized them in two ways:

Scaffolding: Condense verbose thinking into a concise "strategy guide."
Reflect Transformation: The cleverest trick. It doesn't just show the correct path — it explicitly flags: "Watch out, there's a logical trap here that many AIs got wrong before. Don't repeat their mistake."

How Big Is the Impact?

The experimental results are stunning. On the extremely difficult AIME (American Invitational Mathematics Examination) tests:

With thinking-trace retrieval, the mid-tier Gemini 2.5 Flash jumped by 56.3%!
Even the strongest model, GPT-5, gained 7.6% accuracy when inspired by "prior thinking."

Even more remarkable: the method makes the AI not only smarter but also faster and cheaper. With a clear "thinking guide" in hand, the model's own internal reasoning time shortens, reducing computation by about 15%.

Why This Matters for How We Think

Feynman once said: "I have to think from scratch, so that I can understand what I'm doing."

This paper, in a way, proves that the essence of understanding is grasping a "thinking pattern," not memorizing a "dead fact."

We used to try making AI smarter by feeding it more "encyclopedias" — treating it as a "memorization machine." Now, "thinking-trace retrieval" treats it as a "learning machine."

To sum up:

We are entering an era where thinking itself is reusable.

Future AI collaboration may no longer be simple "question and answer," but a vast "relay of thought." When you tackle a hard problem, the wrong turns, avoided pitfalls, and flashes of insight from countless top models worldwide — across time and space — can all become the "scaffolding" inside your own mind.

Don't settle for knowing "what the answer is"; learn "how others thought about it." That may be the ultimate law of co-evolution for humans and machines in the AI era.