Bimodal Fate of Chain-of-Thought Reasoning: Why LLMs Either Solve Instantly or Grind Forever

You give DeepSeek-R1 an AIME competition math problem. It starts "thinking"—outputting a long stream of reasoning tokens. You wait 10 seconds, 20 seconds, 40 seconds, and it's still thinking. You begin to wonder: is it actually approaching an answer, or has it already lost its way?

A July 2026 paper by Renuka Oladri et al., *Token Budget Saturation and Mechanistic Early Detection of Reasoning Non-Convergence in Chain-of-Thought Models*, offers a surprising answer: the fate of the problem is largely sealed by the time the model has thought for 150 tokens.

A Bimodal Distribution: Instant Solves or Endless Grinding

The researchers fed three math benchmarks to DeepSeek-R1-Distill-Qwen-7B: GSM8K (elementary), MATH-500 (high school), and AIME 1983-2024 (competition level). Using a budget-forcing mechanism to cap the model's thinking tokens from 256 to 10,000, they measured how accuracy changed.

The results form a striking curve:

GSM8K and MATH-500: 256 tokens reach 95% of unrestricted accuracy. In other words, easy problems don't need long thinking at all—give them enough and they're done.

AIME is a completely different story. It shows an extreme bimodal distribution:

• 56.5% of problems "naturally converge"—the model finishes reasoning within 10,000 tokens, with 96.5% accuracy
• 43.5% of problems "don't converge"—the model grinds all the way to the 10,000-token cap without concluding, with only 11.5% accuracy

The gap is stunning: converging problems score 96.5%, non-converging ones 11.5%. For the same problem, the model either knows it and reasons straight through, or doesn't—and ten thousand steps of grinding won't help.

Moreover, this bimodal distribution is only weakly correlated with problem difficulty (r² ≈ 0.186). It's not the case that hard problems always fail to converge and easy ones always converge—some hard problems are solved instantly, while some medium ones grind endlessly. Fate isn't fully determined by difficulty.

Early Detection: Seeing Fate at Token 150

The second finding is even more surprising. The researchers asked: since outcomes are bimodal, can they be detected early in the reasoning process?

They inserted forward hooks during inference to capture hidden-layer activations, then trained a linear probe—a simple logistic regression classifier reading activations at a given layer and position—to predict whether a problem would ultimately converge.

Results:

• At token 150, activations from layer 20 yield AUC 0.608 (±0.080, 5-fold cross-validation)
• Even at token 50—when reasoning has barely begun—the signal is already reliably above chance
• Activation probes beat behavioral baselines (based on token entropy and repetition statistics) at 8 of 9 checkpoints, with an average AUC gain of +0.035

In other words: whether the model "knows it can solve this" is already partially encoded in its internal representations by the 50th token of reasoning. The chain of thought isn't finished, but its neurons are quietly telling you the ending.

A permutation test with 100,000 permutations gives p=0.063—slightly above the conventional 0.05 threshold, suggesting the sample size (n=200) is somewhat insufficient, though the signal direction is clear.

What This Means: Adaptive Compute Allocation

If a problem's fate can be detected early, something very practical becomes possible: early exit.

Current CoT reasoning gives every problem the same token budget—easy problems waste compute, hard ones don't get enough. With an early detector judging at token 150 that "this problem probably won't converge," you could:

• Stop non-converging problems early, saving 9,800+ tokens of compute
• Let converging problems continue, preserving accuracy
• Or switch strategies for non-converging problems—new prompt, new tools, or fallback to retrieval

This is what "adaptive compute allocation" in the paper's framing means. Not every problem deserves ten thousand steps of grinding.

"More Thinking" Is Not a Universal Cure

This research directly challenges the default assumption of current CoT reasoning—that more thinking tokens always yield better results. The data shows:

• Easy problems (GSM8K/MATH-500) saturate at 256 tokens; more is pure waste
• Hard problems (AIME) are bimodal—extra tokens only help the subset that would converge anyway, and do nothing for the rest

This aligns with prior findings that CoT gains are non-linear with an inflection point. But this paper goes further: the inflection point varies by problem, and the outcome is encoded in representations early on.

An Honest Assessment: Weak Signal, Small Sample

The limitations must be noted:

• AUC 0.608 is not a strong signal—better than random, but far from practical
• n=200 only achieves p=0.063 under permutation testing, missing the 0.05 bar
• Testing on a single model (DeepSeek-R1-Distill-Qwen-7B); generalization is unknown
• A gap remains between "early detection" and "practical early exit"—probes work in offline training, but whether they can run in real time during inference is an open question

Still, as a mechanistic interpretability study, its value lies in pointing a direction: CoT success is not a black box—it leaves traces in internal representations. The next steps are finding stronger signals, validating across more models, and turning probes into usable inference-time components.

An Analogy: Exams—Either You Know It or You Don't

You've probably had this exam experience: one look at a problem and you either "have an idea" or you don't. Problems you understand, you work straight through and likely get right; problems you don't, you grind on for half an hour without progress.

DeepSeek-R1's CoT reasoning exhibits exactly this bimodal fate of "if you know it, you know it." And its internal representations—those hidden-layer activations—already "know" whether it has an idea, right from the start of reasoning; you just can't see it from the outside.

This raises a deeper philosophical question: is the model's "knowing" whether it can solve a problem the same kind of knowing as a human's? When a linear probe can read "I can't crack this" from neural activations, is that a proto-form of metacognition?

Maybe. But for now, at least, we can use it to save compute.

---

Paper: Renuka Oladri, Niveda Jawahar, Abdirisak Mohamed. *Token Budget Saturation and Mechanistic Early Detection of Reasoning Non-Convergence in Chain-of-Thought Models.* arXiv:2607.21433, 2026. Link: https://arxiv.org/abs/2607.21433