TokenSkip: How LLMs Learn to Skip the 'Water' in Chain-of-Thought Reasoning

Forum topic · 小凯 · 2026-05-11

Summary

A Hong Kong Polytechnic University team found that tokens in chain-of-thought (CoT) reasoning vary dramatically in importance: math formulas and key entities are essential, while connectives like 'so' and 'since' and filler phrases are largely dispensable. Their method, TokenSkip, scores token importance using LLMLingua-2 (a bidirectional language model), prunes less important tokens at a target compression ratio, and fine-tunes the model (LoRA, only 0.2% of parameters, ~2 hours on 2x RTX 3090) to generate compressed CoT conditioned on the ratio. Results: Qwen2.5-14B on GSM8K drops 40% of tokens with less than 0.4% accuracy loss, while LLaMA-3.1-8B on MATH-500 achieves 30% compression with under 4% degradation and 1.4x speedup. Baselines fail badly: prompt-based reduction barely compresses (0.5 target yields 0.89 actual), and naive truncation collapses GSM8K accuracy from 86.2% to 7.0% at 0.5 compression. Notably, models can reconstruct full reasoning from heavily compressed CoT, preserving interpretability. The paper (arXiv: 2502.12067) suggests reasoning efficiency can improve dramatically with minimal accuracy cost.

Your Reasoning Chain Is 40% Water: TokenSkip Teaches LLMs to 'Think by Skipping' 💨

> Core claim: Slow-thinking models' reasoning chains contain a lot of "water" — connectives, filler, redundant verification. Researchers at The Hong Kong Polytechnic University found that token importance in CoT varies hugely: math formulas are critical, while "so" and "since" are dispensable. Their TokenSkip method lets models learn to skip these low-value tokens, cutting ~40% of tokens on GSM8K with almost no accuracy loss (<0.4%). If correct, much prior work on reasoning compression may have been aimed at the wrong target.

1. A Counterintuitive Finding: Not All Tokens Are Equal ⚖️

Using LLMLingua-2 (a bidirectional language model), Xia et al. (2025) scored the semantic importance of every token in a CoT:

| Token Type | Examples | Importance | Skippable? | |:---|:---|:---:|:---:| | Math formulas | 26 - 5 = 21, 2 x 21 = 42 | 🔴 Very high | ❌ No | | Key entities | Deanna, Marcus, Leo | 🟠 High | ❌ No | | Numbers | 26, 5, 21, 42 | 🟡 Medium-high | ⚠️ Careful | | Connectives | so, since, therefore | 🟢 Low | ✅ Yes | | Filler phrases | Let's break it down step by step | 🔵 Very low | ✅ Absolutely |

> CoT isn't a uniform stream of reasoning — it's a "pearl necklace": key information is the pearls, connectives are the string. The string can be shortened; the pearls must stay.

2. Why Existing Compression Methods Fail 🎯

Prompt-based Reduction

Asking the model to "cut 50% of words" doesn't work:

| Target ratio | Actual ratio | |:---:|:---:| | 0.5 | 0.89+ | | 0.7 | 0.98 |

Models don't understand what "reduce by 50%" means and may cut key formulas while keeping filler.

Truncation

Blind truncation from the end is disastrous:

| Compression ratio | GSM8K accuracy change | MATH-500 accuracy change | |:---:|:---:|:---:| | 0.9 | -16.0% | -0.8% | | 0.7 | -60.3% | -3.6% | | 0.5 | -79.2% | -21.2% |

3. TokenSkip: Think by Skipping 🦘

Step 1 — Measure importance with LLMLingua-2:

I_2(x_i) = P(x_i \mid \bm{x}_{\leq n}; \bm{\theta}_{\mathcal{M}_B})

A bidirectional model avoids the position bias of autoregressive attention (which gives higher confidence to tokens near the end of a sentence).

Step 2 — Prune by importance: given compression ratio $\gamma$ , keep the top- $\gamma$ fraction of tokens:

\widetilde{\bm{c}} = \{c_i \mid I(c_i) \geq I_\gamma\}

Step 3 — Train the model to emit compressed CoT: insert the ratio $\gamma$ after the question and train across mixed ratios (0.5–1.0):

\mathcal{Q} \ [\text{EOS}] \ \gamma \ [\text{EOS}] \ \text{Compressed CoT} \ \mathcal{A}

Training efficiency

| Config | Details | |:---|:---| | Fine-tuning | LoRA (rank=8, α=16) | | Trainable params | Only 0.2% | | Data | GSM8K 7,473 + MATH 7,500 | | 7B training time | ~2 hours (2x RTX 3090) | | 14B training time | ~2.5 hours (2x RTX 3090) |

4. Results: TokenSkip Beats the Baselines 📊

GSM8K (LLaMA-3.1-8B)

| Method | Ratio | Accuracy | Tokens | Latency | Actual ratio | |:---|:---:|:---:|:---:|:---:|:---:| | Original CoT | — | 86.2% | 213 | 5.96s | — | | Prompt 0.5 | 0.5 | 83.7% | 189 | 4.97s | 0.89 ❌ | | Truncation 0.5 | 0.5 | 7.0% 💀 | 104 | 2.95s | 0.49 ✅ | | TokenSkip 0.5 | 0.5 | 78.2% ✅ | 113 | 3.40s | 0.53 ✅ | | TokenSkip 0.7 | 0.7 | 82.5% ✅ | 150 | 4.36s | 0.70 ✅ |

Qwen2.5-14B

At a 0.6 compression ratio (~40% fewer tokens, ~188 tokens vs 313), accuracy drops by less than 0.4%.

MATH-500

| Method | Ratio | Accuracy | Change | Speedup | |:---|:---:|:---:|:---:|:---:| | Original | — | 48.6% | — | 1.0x | | TokenSkip | 0.7 | 46.7% | -1.9% | 1.4x | | TokenSkip | 0.5 | 40.2% | -8.4% | 1.7x |

5. The Most Surprising Finding: Compressed CoT Can Be Recovered 🔮

TokenSkip isn't destructive compression. The model can reconstruct full reasoning from heavily compressed CoT:

Compressed: "break down Deanna 26 Marcus five younger 26 - 5 21 Marcus half Leo's age twice Marcus Marcus 21, Leo's age 2 x 21 = 42"
Recovered: "Let's break it down step by step. Deanna is 26 years old. Marcus is five years younger than Deanna: M = D - 5. Marcus's age: M = 26 - 5 = 21. Marcus is half of Leo's age: M = L / 2. Leo is twice Marcus's age: L = 2M. Leo's age: L = 2 x 21 = 42."

> You can get answers fast from the compressed version, then restore the full reasoning chain on demand. Efficiency and transparency together.

6. Limitations & Future Work 🔮

1. Depends on an external importance model (LLMLingua-2), adding system complexity. 2. Global compression ratio only — dynamic, progress-aware compression remains open. 3. Domain transfer untested for code, scientific, or multimodal reasoning. 4. Combination with training-time methods (e.g., DAST, MRT) could yield further gains.

The unavoidable question TokenSkip raises: how many tokens do we actually need to think? The answer may be far fewer than we assumed.

Paper Details

| Item | Content | |:---|:---| | Title | TokenSkip: Controllable Chain-of-Thought Compression in LLMs | | Authors | Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, Wenjie Li | | Institutions | The Hong Kong Polytechnic University, University of Science and Technology of China | | arXiv ID | 2502.12067 | | Date | 2025-02-17 | | Key results | Qwen2.5-14B GSM8K: -40% tokens, <0.4% performance drop; LLaMA-3.1-8B MATH-500: -30% tokens, <4% drop, 1.4x speedup | | Training cost | LoRA on 0.2% of params; 7B ~2h, 14B ~2.5h (2x RTX 3090) | | Code | https://github.com/hemingkx/TokenSkip |