LEASH: Training-Free Adaptive Early Stopping for Efficient Chain-of-Thought Reasoning

Forum topic · ✨步子哥 · 2025-11-08

Summary

LEASH (Logit-Entropy Adaptive Stopping Heuristic), proposed by Quamar and Areeb (arXiv:2511.04654), is a training-free, plug-and-play algorithm that adaptively halts chain-of-thought (CoT) reasoning in large language models. During decoding, LEASH monitors two internal signals: the slope of token-level entropy over a sliding window and the improvement of the top-logit margin between the two most likely tokens. When both signals plateau, and after passing a triple-gated stop rule (minimum warm-up length, sufficient entropy drop, and majority voting over recent non-saturated steps), the model stops generating rationales and directly requests the final answer. Experiments across Llama-3.1-8B-Instruct, Mistral-7B-v0.1, Phi-3-Mini-128k-Instruct, and Qwen2.5-7B-Instruct on GSM8K and AQuA-RAT show consistent 28-42% token reductions and ~28-30% latency gains, at a cost of roughly 9-13 accuracy points versus full CoT, while still outperforming No-CoT baselines. The O(1) per-step overhead makes LEASH easy to integrate, offering a practical accuracy-efficiency trade-off for cheaper, faster LLM inference and greener AI.

Overview

Chain-of-Thought (CoT) prompting boosts LLM reasoning on math and logic tasks, but generates verbose rationale chains: an estimated 30-40% of tokens are redundant "overthinking," inflating latency by about a third. Existing fixes (fixed-length truncation, keyword heuristics, multi-sample reranking) are rigid or compute-hungry.

LEASH (Logit-Entropy Adaptive Stopping Heuristic) from Quamar & Areeb (arXiv:2511.04654) is a training-free, plug-and-play adaptive stopping method that lets the model itself decide when its reasoning has converged.

Key points

Two monitored signals:
Token-level entropy slope: $s_H(t; k) = \frac{H_t - H_{t-k}}{k}$ over a window of $$k=8$$ steps; near-zero slope (within $\epsilon_H=0.005$ ) signals convergence.
Top-logit margin change: $\Delta M(t; k) = M_t - M_{t-k}$ where $M_t = \ell^{(1)}_t - \ell^{(2)}_t$ ; improvement below $\delta_M=0.05$ signals confidence saturation.
Saturation filtering: steps with peak probability $p_{\text{max}}(t) \geq \tau_p$ (e.g., punctuation, stop words) are excluded from trend analysis to avoid misleading signals.
Triple-gated stop rule: (1) exceed minimum warm-up length $t_{\text{min}}$ ; (2) sufficient total entropy drop $H_{\text{ref}} - H_t \geq \gamma$ ; (3) at least half of the last $$L=5$$ non-saturated steps pass the plateau test (majority voting). Increasing $$L$$ from 3 to 5 cut premature stops by 40% in ablations.
O(1) overhead: implemented with a ring buffer; adds only a few floating-point operations per step. Works with INT8 quantization (31% token reduction on Mistral-7B preserved).

Experimental results

Tested on GSM8K (300 problems) and AQuA-RAT across four open-source models:

Llama-3.1-8B-Instruct: tokens cut from ~320 to ~220 (-31%); latency per problem 4.04s → 2.84s (-29.7%); accuracy 74.33% → 62.32%.
Phi-3-Mini-128k-Instruct: ~290 → ~170 tokens (-41.5%), the largest reduction; accuracy 82.67% → 69.87%.
Mistral-7B-v0.1: 35.12% token reduction; baseline accuracy 47.20%.
Qwen2.5-7B-Instruct: smallest accuracy loss (~9.18 points); on AQuA-RAT, 28.15% token reduction matched by 28.10% latency reduction.
Averages: ~10.9 accuracy points traded for ~35.3% token savings on GSM8K; LEASH always beats the No-CoT baseline.

Analysis and implications

Errors fall into two categories: "premature stops" on problems needing long chains, and "stable hallucinations" where plateau signals occur on a wrong reasoning path. Full CoT itself contains 15-20% redundant content, framing the trade-off as reasonable.
Saved compute can be "reinvested": the same hardware serves ~43% more requests at 30% token reduction, or funds answer verification / multi-path fusion to recover accuracy.
Future directions: long-form generation, tool-augmented reasoning, theoretical stopping guarantees for safety-critical use, multimodal extension, and green-AI impact (estimated 15-20% inference energy savings if widely adopted).

Reference

Quamar, M. A., & Areeb, M. (2025). Logit-Entropy Adaptive Stopping Heuristic for Efficient Chain-of-Thought Reasoning. arXiv:2511.04654.