LEASH: Training-Free Adaptive Early Stopping for Efficient Chain-of-Thought Reasoning

Forum topic · ✨步子哥 · 2025-11-08

Summary

LEASH (Logit-Entropy Adaptive Stopping Heuristic) is a training-free, plug-and-play algorithm that lets large language models decide when their chain-of-thought (CoT) reasoning has converged, cutting wasted tokens without any retraining. By monitoring two internal decoding signals—the slope of token-level entropy and the improvement of the top-logit margin—LEASH detects when a model's rationale generation shifts from exploration to redundant repetition, then stops the rationale and requests the final answer. A saturation filter excludes overconfident steps (e.g., punctuation), and a majority-vote gate over recent non-saturated steps prevents premature stopping. On GSM8K across four open models (Llama-3.1-8B-Instruct, Mistral-7B, Phi-3-Mini, Qwen2.5-7B), LEASH reduced token generation by 30-40% and latency by ~28-30%, with an average accuracy cost of about 10 percentage points versus full CoT (e.g., Llama-3.1-8B: 74.33% to 62.32%, 31% fewer tokens, 4.04s to 2.84s per question). Results generalize to AQuA-RAT and remain robust under INT8 quantization. The paper (arXiv:2511.04654) positions LEASH as a practical step toward greener, more economical LLM inference.

Overview

Chain-of-Thought (CoT) prompting boosts LLM performance on math and logic tasks, but it comes at a steep cost: models generate lengthy, often redundant rationales. The paper *Logit-Entropy Adaptive Stopping Heuristic for Efficient Chain-of-Thought Reasoning* (Quamar & Areeb, 2025, arXiv:2511.04654) introduces LEASH, a training-free, plug-and-play stopping algorithm that teaches AI when to "brake" its own thinking.

Key points

The problem: Standard CoT produces over-thinking on easy problems, wasting 30-40% extra tokens and inflating latency by roughly a third. Existing fixes are crude: fixed-length truncation, keyword-based heuristics, or costly multi-sample re-ranking.
Core idea: Monitor two intrinsic decoding signals—
Token-level entropy slope: $s_H(t; k) = \frac{H_t - H_{t-k}}{k}$ over a window of $$k=8$$ steps; near-zero slope ( $\epsilon_H=0.005$ ) signals the model has converged.
Top-logit margin improvement: $\Delta M(t; k) = M_t - M_{t-k}$ where $M_t = \ell^{(1)}_t - \ell^{(2)}_t$ ; improvement below $\delta_M=0.05$ indicates confidence saturation.
Safety mechanisms: A saturation filter marks steps with peak probability $p_{\text{max}}(t) \geq \tau_p$ (e.g., punctuation tokens) and excludes them from trend analysis. Stopping requires: minimum warm-up length $t_{\text{min}}$ , sufficient entropy drop $H_{\text{ref}} - H_t \geq \gamma$ , and at least half of the last $$L=5$$ non-saturated steps passing stationarity tests (majority voting cut mis-stops by 40% in ablations).
Near-zero overhead: O(1) per-step cost via a ring buffer; negligible versus transformer forward passes; remains effective under INT8 quantization (31% token reduction on Mistral-7B).

Experimental results (GSM8K, 300-problem subset)

| Model | Baseline CoT acc. | LEASH acc. | Token reduction | Latency gain | |---|---|---|---|---| | Llama-3.1-8B-Instruct | 74.33% | 62.32% | ~31% (320→220) | 29.7% (4.04s→2.84s) | | Mistral-7B-v0.1 | 47.20% | — | 35.12% | — | | Phi-3-Mini-128k | 82.67% | 69.87% | 41.50% (~290→170) | — | | Qwen2.5-7B-Instruct | 65.33% | — | ~28-31% | 28.10% |

Token reduction stays consistently in the 30-40% "golden range" across architectures.
On AQuA-RAT (harder algebra), Qwen2.5-7B showed 28.15% token reduction and 28.10% latency reduction, confirming robustness across difficulty distributions.
Average accuracy trade-off: ~10.9 percentage points sacrificed for ~35.3% token savings—LEASH always outperforms the No-CoT baseline.

Analysis

Error modes: "Premature stops" on problems needing long chains, and "stable hallucinations" where entropy/margin appear flat but the model is stuck in circular reasoning.
Model differences: Phi-3-Mini's high reduction rate (41.5%) suggests noisy rationales; Qwen2.5-7B's smallest accuracy drop (9.18 points) suggests more compact reasoning.
Reinvestment argument: 30% token savings mean ~43% more requests on the same hardware, or budget for answer verification to offset accuracy loss.

Limitations and future work

Currently assumes short, verifiable answers; extension to open-ended generation, tool-augmented reasoning, and multimodal (vision-language) decoding is proposed.
Lacks theoretical stopping guarantees; proving safety bounds for safety-critical applications is a key open direction.
Wide adoption could cut LLM inference energy consumption by an estimated 15-20%.

References

1. Quamar, M. A., & Areeb, M. (2025). Logit-Entropy Adaptive Stopping Heuristic for Efficient Chain-of-Thought Reasoning. *arXiv preprint arXiv:2511.04654*. 2. Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. *NeurIPS*. 3. Chen, M., et al. (2023). Teaching Models to Express Their Uncertainty in Words. *ICLR*. 4. Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. *ICLR*. 5. Zhang, Z., et al. (2024). Adaptive Chain-of-Thought Decoding via Confidence-based Early Stopping. *ACL*.