The Mic Is Still Hot, but the Answer Has Gone Cold — OTBench Draws a Braking-Distance Curve for Overthinking

*Translation of a zhichai.net forum post reviewing "Constructing Pareto Frontiers of Optimal Reasoning Termination Methods for Overthinking in Reasoning Models" (arXiv: 2610.13570).*

One-line summary

Researchers from the Institute of Information Engineering, Chinese Academy of Sciences and collaborators built a "ruler for thinking" (STP) and ran over 90,000 mid-reasoning truncation experiments on 92,000+ reasoning traces from six models. Two stark numbers emerged: on average 64.3% of a reasoning model's thinking tokens are produced after accuracy has already peaked (pure waste), and full-length reasoning is on average 9.03 percentage points worse than stopping at the optimal point (not just wasted — actively harmful). They then place the three mainstream stop-thinking strategies in one mathematical framework, derive each one's theoretical limit — its Pareto frontier — and find that all existing methods run below their limits. The measurement infrastructure and benchmark are open-sourced as OTBench.

When "thinking longer" no longer means "answering better"

Since late 2024, reasoning models (from OpenAI's o1 to DeepSeek-R1) have dominated AI: before answering, they emit long chains of reasoning, self-checks, and corrections. The industry consensus — more thinking, better answers — has inflated reasoning budgets from thousands to hundreds of thousands of tokens. But reasoning tokens are money: GPU hours, latency, energy. The entire cost structure rests on an unaudited assumption that thinking is worth it.

Intuition says returns on thinking must have an asymptote. Past some point, more thought dilutes rather than refines the answer. But before OTBench, three questions had no numbers:

1. How much redundant thinking is there — 10%? 50%? 2. Does extra thinking actually flip correct answers to wrong ones? 3. Which stopping strategy — per-question routing, fixed budgets, or per-trace early exit — has the highest theoretical ceiling?

The paper turns these feelings into measurements in two steps: build a ruler (STP), then draw a map (unified Pareto frontiers for termination strategies).

STP: pulling the speech away at page k

STP (Step-wise Truncation Probing) works like an editor cutting a manuscript: split each reasoning trace into steps of ~256 tokens (cutting at blank lines so each step is semantically coherent), truncate at step k, and let the model finish the answer from there — independently generated 16 times per cut point, taking the majority vote to suppress randomness. Sweeping k from 1 to n yields a token–accuracy response curve.

Two orthogonal metrics fall out:

• Overthinking Ratio (OR): the fraction of tokens generated after the earliest point k* achieving peak accuracy: 1 − L(k*)/L(n). Meeting analogy: how long the meeting drags on after the decision is made.
• Accuracy Degradation (AD): P(k*) − P(n), how much accuracy drops from the best point to the end of reasoning. Analogy: how much the final conclusion is worse than the best moment's.

Scale: 963 problems (AIME 90, MATH500 500, GPQA-Diamond 198, LiveCodeBench 175) × 16 traces each = 92,448 traces, covering GPT-OSS-20B, Qwen3-4B-Thinking, Qwen3-30B-A3B-Thinking, and DeepSeek-R1-Distill-Qwen-7B/14B/32B. Total: 70.5B reasoning tokens of probing data — a full CT scan of six models' thinking.

The measured disease: 60% wasted, 10% worsened

• Average OR = 64.3% — over six in ten thinking tokens are generated after accuracy peaks.
• Average AD = 9.03pp — full reasoning is on average ~9 points worse than stopping at the optimal point. Overthinking isn't just waste; it's negative return.
• Across models: OR ranges 41.1%–74.4%; AD ranges 5.64–13.40pp.

Two striking per-benchmark extremes:

• MATH: "verbose but harmless" — OR 78.8%, AD only 1.83pp. Late-stage math reasoning mostly re-verifies already-confirmed steps.
• GPQA: "brief but lethal" — OR 61.4%, AD 16.98pp (the highest, ~10× MATH). Late scientific reasoning is overconfident self-doubt that flips correct answers.

These extremes underpin a key statistical finding: OR and AD correlate at only Spearman −0.330 — redundancy and harm are two independent diseases needing two independent cures. Any efficiency fix that only shortens thinking may cut harmless verification on math or miss fatal corrections on GPQA.

Even on the same problem, the 16 sampled traces differ wildly: average absolute trace-to-trace differences are 20.23pp for OR and 10.06pp for AD. A uniform per-problem budget is doomed by the data alone. Micro-level observation: late reasoning steps are mostly "no change" (circling, restating, re-verifying), and the ratio of harmful to helpful changes worsens as reasoning proceeds.

Three brakes, one racetrack map

The paper frames "when to stop thinking" as action selection under a budget, with three strategies:

• Route A — Dynamic Routing (DR): per-problem binary choice (0 = no thinking, 1 = full thinking). E.g., ThinkSwitcher.
• Route B — Fixed Budget (FB): per-problem token cap. E.g., Predictive Scheduling.
• Route C — Dynamic Early Exit (DEE): per-trace stopping decision. E.g., ThinkBrake.

They nest cleanly: DR ⊆ FB ⊆ DEE — flexibility equals theoretical potential. Corresponding Pareto frontiers satisfy F_DR(B) ≤ F_FB(B) ≤ F_DEE(B) (budget on the x-axis, accuracy on the y-axis; inside the frontier is waste, outside is fantasy).

How big are the frontier gaps?

To match full reasoning accuracy (A_full = 72.33%):

• DR needs 71.4% of the budget — barely saves anything.
• FB needs 35.0% — a qualitative leap.
• DEE needs 14.2% — under one-seventh of the budget.

Accuracy ceilings: DR ~73%, FB ~76%, DEE ~81%. Even with unlimited budget, DR tops out near full reasoning; DEE's ceiling exceeds full reasoning by ~9 points — complete thinking isn't even the theoretical runner-up. And DEE's ceiling comes cheap: only 58.6% of budget needed (DR: 94.2%, FB: 87.2%).

A pleasing consistency: DEE's 81% ceiling minus full reasoning's 72.33% ≈ the 9.03pp average AD from STP — measurement and theory independently point to the same pathology.

Caveats the authors flag: the signal-computation overhead of "should I stop?" is not counted in the budget, so DEE's theoretical advantage is a non-discounted reserve.

Racing toward the limit: every driver still has fuel

All existing public methods sit below their own strategy's theoretical frontier: ThinkSwitcher under the DR frontier, Predictive Scheduling under FB, ThinkBrake under DEE — gaps from a few to a dozen-plus points. The unit of comparison changes: no longer "X points above SOTA," but "how close to your strategy class's physical limit." The gap below the frontier is now the field's coordinate system for progress.

OTBench open-sources the infrastructure:

• 70.5B reasoning tokens of probing measurements;
• public train/validation/test splits;
• a unified strategy interface;
• dedicated evaluators;
• six models, four benchmark suites.

OTBench is not another leaderboard — it is a contoured map: leaderboards tell you who's on the hill; the map tells you which path can theoretically reach the summit.

Stated limitations: only six models and four benchmark suites (generalization to larger flagship models untested); STP is a diagnostic tool, not an online stopping algorithm (16 probes per cut point make it radiology, not pharmacy); Pareto frontiers are offline, "god's-eye-view" references, not deployable policies; and stopping-signal overhead is uncounted. Each boundary is an entry point for follow-up work — the mark of a good benchmark: it ends one debate (how severe is overthinking) and opens ten new questions.

Back to the series: why benchmarks need braking points

This is the third piece in the "vitality of evaluation" series. Where previous installments covered defense (aging targets, overfitting), OTBench shows evaluation's offense: benchmarks can measure a model's *process* and identify which segment of it is wasted labor. Overthinking was once an adjective; now it is 64.3% and 9.03pp. The sharpest cut: every racer is currently competing far from the finish line's true contest — who first touches the physical limit. The benchmark has turned from scoreboard into starting gun.

Key numbers

| Metric | Value | |---|---| | Average Overthinking Ratio (OR) | 64.3% | | Average Accuracy Degradation (AD) | 9.03pp | | OR range across models | 41.1% – 74.4% | | AD range across models | 5.64 – 13.40pp | | MATH: OR / AD | 78.8% / 1.83pp | | GPQA: OR / AD | 61.4% / 16.98pp | | OR–AD Spearman correlation | −0.330 | | Trace-to-trace OR / AD avg. abs. diff. | 20.23pp / 10.06pp | | Budget to match 72.33% accuracy | DR 71.4% → FB 35.0% → DEE 14.2% | | Accuracy ceilings | DR ~73% / FB ~76% / DEE ~81% | | Budget to reach ceiling | 94.2% / 87.2% / 58.6% | | Scale | 963 problems × 16 traces = 92,448 traces, 70.5B tokens | | STP parameters | step size b = 256 tokens, 16 probes per position |

Reference

Constructing Pareto Frontiers of Optimal Reasoning Termination Methods for Overthinking in Reasoning Models Weixin Guan, Liang Li, Shiyu Xie, Jiapeng Liu, Bing Li, Peng Fu, Chengyang Fang, Xiaoshuai Hao, Can Ma, Weiping Wang Institute of Information Engineering (CAS), UCAS, University of Florida, Jiangxi University of Finance and Economics, Li Auto arXiv: 2610.13570 (2026-10-09)