The Coupling Tax: Why Longer Chain-of-Thought Can Make LLMs Worse Under Shared Token Budgets
Core claim: Nie et al. (2026) identify an overlooked cost — the "Coupling Tax". When a reasoning chain and the final answer are forced to share the same token budget, the more a model "thinks," the less room remains for the answer. On GSM8K and MATH-500, non-thinking mode scores higher than long-thinking mode within 2048 tokens. Long reasoning chains only pay off when tasks are hard enough and budgets large enough. The authors propose split-budget generation with independent token pools for reasoning and answer — reaching 83.6% on MATH-500. If correct, current visible-CoT interface designs are fundamentally flawed.
1. A Zero-Sum Game: Thinking vs. Answering
Conventional wisdom says "let the model think more" — longer CoT, more test-time compute. But Nie et al. ask an unasked question:
> When the reasoning chain and answer share one output budget, they compete for the same cake.
It's like taking an exam where your scratch paper is the answer sheet: the more scratch work, the less space for the final answer.
2. Experiments: Non-Thinking Wins?
Comparing Qwen3 thinking vs. non-thinking modes:
| Task | Budget | Thinking | Non-thinking | Winner | |:---|:---:|:---:|:---:|:---:| | GSM8K | ≤2048 tokens | baseline | matches or exceeds | non-thinking | | MATH-500 | ≤2048 tokens | baseline | matches or exceeds | non-thinking | | BIG-Bench Hard | small budget | baseline | possibly better | non-thinking | | BIG-Bench Hard | large budget | exceeds | baseline | thinking |
The crossover budget shifts with task difficulty: on GSM8K (easy), non-thinking always wins; on MATH-500 (medium), thinking wins only above 2048 tokens; on BIG-Bench Hard (hard), an even larger budget is needed. Fixed thinking budgets across all tasks therefore constitute serious resource misallocation.
3. Mathematical Explanation: Truncation-Waste Decomposition
The paper offers a decomposition:
where \(b\) is the total token budget, \(\alpha_c\) is accuracy when the chain is truncated, \(\alpha_t\) is accuracy after complete thinking, and \(F_L(b)\) is the CDF of chain lengths ≤ \(b\).
- Small \(b\): many chains truncated → accuracy near low \(\alpha_c\)
- Large \(b\): few truncations → accuracy near higher \(\alpha_t\)
- TokenSkip: 40% of CoT tokens are "water." Combined: compress CoT first, then split budgets — shorter CoT plus more complete answers.
- DAST: difficulty-adaptive thinking length is not merely an optimization but a necessity to avoid the coupling tax.
- 80/20 rule: high-entropy tokens steer reasoning; truncation may cut off the most critical decision points.
- Fixed split ratios (e.g., 50/50) vs. dynamic allocation per question type
- Whether RLVR training with separated budgets yields more compact reasoning
- Whether the coupling tax is amplified in multi-turn settings where context consumes budget
- Whether large models need larger minimum budgets to show their advantage
This also explains inverse scaling in the Qwen family: larger models generate longer chains, thus face higher truncation probability under fixed budgets — so bigger models can perform worse.
4. Solution: Split Budgets
Current visible-CoT interfaces share one budget across system prompt, thinking, and answer. The proposed split-budget generation (IRIS) gives reasoning and answer independent token pools:
| Configuration | MATH-500 accuracy | |:---|:---:| | Baseline (shared budget) | baseline | | IRIS (split budget) | 74.0% | | Reinforced extraction variant | 78.8% | | Fixed non-oracle SC+IRIS gate | 83.6% |
The SC+IRIS gate combines Self-Consistency (multiple samples, majority vote) with IRIS, and dynamically decides whether thinking mode is needed based on question difficulty — without knowing the correct answer (non-oracle). The 83.6% result is achieved without increasing total compute — the budget is merely reallocated.
5. Cross-Model Validation
The same pattern reproduces on DeepSeek-R1-Distill-Llama-8B. The coupling tax is not a bug of one model or interface but a systemic flaw of the shared-budget design paradigm.
6. Connections to Prior Work
7. The Bet
The author bets $1,000: by end of 2026, all mainstream visible-CoT models (DeepSeek, Qwen, Kimi, etc.) will adopt some form of budget-split design. Reasons: the gains are strong, the change requires no retraining (interface-level only), the theory reframes test-time reasoning as a budget-allocation optimization, the effect is universal across architectures, and users care about complete answers, not visible thinking.
8. Limitations and Future Work
Paper Details
| Item | Content | |:---|:---| | Title | The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits | | Authors | Wenhua Nie, Junlin Liu, Jianan Wu, Zijie Meng, Yilong Fan, Zhang Zijian, Haoran Zheng, Jyh-Shing Roger Jang | | arXiv ID | 2605.07686 | | Date | 2026-05-08 | | Key results | Non-thinking mode matches/exceeds thinking mode on GSM8K/MATH-500 within 2048 tokens; IRIS 74.0% → 83.6% on MATH-500; confirmed on Qwen3 and DeepSeek-R1-Distill-Llama-8B |