Have you ever taken an exam where the question was so long that by the time you reached the end, you'd forgotten what was said at the beginning?
Large language models have the same problem. Researchers have even given it a vivid name: Lost-in-Thought.
Here's the background. Today's reasoning models are powerful—they work through problems step by step, laying out many steps like solving a math problem. In theory, the more steps, the more thorough the thinking, right?
But researchers found a surprising pattern: as the model's reasoning chain grows longer, its ability to find evidence from the context actually declines.
It's like a student who writes three full pages of scratch work to solve a problem thoroughly—then, on page three, suddenly forgets that the conditions on page one clearly stated "x must be positive."
Reasoning and retrieval, it turns out, are not naturally harmonious. A longer reasoning chain consumes the model's attention on the original context. Thinking for too long makes the model forget why it started thinking in the first place.
The Solution: RecaLLM
The paper proposes a method called RecaLLM. Its core idea is simple: don't let the model think all the way through in one pass. Instead, have it stop repeatedly during reasoning and explicitly retrieve the evidence it needs from the original text.
Imagine reading a long legal document. An ordinary person might read it start to finish and answer questions from memory. But a good lawyer marks things up while reading and flips back to earlier clauses to verify. RecaLLM teaches the model this "retrieve while reasoning" workflow.
Even better, the researchers designed a nearly zero-overhead constrained decoding mechanism that makes the model copy evidence verbatim from the source text during retrieval. This avoids the common failure mode where models embellish or outright fabricate content when "quoting."
A Surprising Training Result
The training data is also interesting. The authors didn't use ultra-long texts of 100K+ tokens—they used samples no longer than 10K tokens. Yet in testing, the model stably handled contexts of up to 128K tokens.
What does this mean? The bottleneck for long-context ability may not be "how long a text the model has seen," but whether it has learned the habit of actively flipping back to the source while thinking.
Takeaway
This finding is deeply inspiring. We often imagine AI capability growth as a one-way process of "more data, bigger models." But this paper shows that sometimes the bottleneck is not hardware or data volume—it's workflow design.
Thinking longer isn't as valuable as thinking smarter. Knowing when to stop and check the source—that is real intelligence.
---
Paper info
- Title: RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval
- arXiv: 2604.09494
- Core finding: Increasing reasoning steps degrades a model's ability to retrieve evidence from context; the fix is interleaving reasoning with explicit retrieval.