TokenSkip: Controllable Chain-of-Thought Compression Cuts 40% of Reasoning Tokens with Minimal Accuracy Loss

Forum topic · 小凯 · 2026-05-11

Summary

Researchers at The Hong Kong Polytechnic University propose TokenSkip, a method for controllable chain-of-thought (CoT) compression in large language models. Their analysis using LLMLingua-2 shows that tokens in reasoning chains vary greatly in importance: math formulas and key entities are critical, while connectives like "so" and "since" and filler phrases are largely dispensable. TokenSkip prunes low-importance tokens at a target compression ratio, then fine-tunes the model (LoRA, only 0.2% of parameters, ~2 hours on two RTX 3090s) to generate compressed CoT directly, conditioned on the ratio. Experiments show Qwen2.5-14B loses less than 0.4% accuracy on GSM8K while dropping 40% of tokens, and LLaMA-3.1-8B achieves 1.4x speedup on MATH-500 with under 4% accuracy decline. The approach also preserves recoverability: models can reconstruct full reasoning from heavily compressed chains. Compared to prompt-based reduction (which fails to compress) and truncation (which collapses accuracy), TokenSkip offers practical, low-cost inference efficiency gains. Paper: arXiv 2502.12067; code at github.com/hemingkx/TokenSkip.

Your Reasoning Chain Is 40% Water: TokenSkip Teaches LLMs to "Think in Jumps"

> Core claim: Slow-thinking models produce reasoning chains full of "water" — connectives, filler, repeated verification. A team from The Hong Kong Polytechnic University found that token importance in Chain-of-Thought (CoT) varies dramatically: math formulas are critical, while "so" and "since" are skippable. Their method, TokenSkip, trains models to skip these water tokens — cutting 40% of tokens on GSM8K with almost no accuracy loss (<0.4%). If correct, much of the past work on reasoning compression may have been headed in the wrong direction.

---

1. A counterintuitive finding: not all tokens are created equal

Consider a reasoning snippet:

> "Let's break it down step by step. 1. Deanna is 26 years old. 2. Marcus is five years younger than Deanna, so Marcus is 26 - 5 = 21 years old. 3. Marcus is half of Leo's age, so Leo's age is twice Marcus's age. 4. Since Marcus is 21, Leo's age is 2 x 21 = 42."

Xia et al. (2025) used LLMLingua-2 (a bidirectional language model) to score each token's semantic importance in CoT:

| Token type | Examples | Importance | Skippable? | |:---|:---|:---:|:---:| | Math formulas | 26 - 5 = 21, 2 x 21 = 42 | Very high | ❌ No | | Key entities | Deanna, Marcus, Leo | High | ❌ No | | Numbers | 26, 5, 21, 42 | Medium-high | ⚠️ Careful | | Connectives | so, since, therefore | Low | ✅ Yes | | Filler | Let's break it down step by step | Very low | ✅ Yes |

> Conclusion: CoT is not a uniform reasoning stream but a "pearl necklace" — key information is the pearls, connectives are the string. The string can be shortened; the pearls must be kept.

---

2. Why existing compression methods fail

Method 1: Prompt-based reduction

Asking the model to "reduce output by 50%" doesn't work:

| Target ratio | Actual ratio | Problem | |:---:|:---:|:---| | 0.5 | 0.89+ | Model doesn't comply | | 0.7 | 0.98 | Almost no compression |

The model doesn't understand what "50% fewer words" means; it may delete key formulas while keeping filler.

Method 2: Truncation

Directly limiting max output length is destructive:

| Ratio | GSM8K accuracy change | MATH-500 accuracy change | |:---:|:---:|:---:| | 0.9 | -16.0% | -0.8% | | 0.7 | -60.3% | -3.6% | | 0.5 | -79.2% | -21.2% |

Truncation is blind: it cuts from the end, while critical information is often in the middle and end of the chain.

---

3. TokenSkip: skipping the unimportant

> If we know which tokens matter, why not train the model to skip the rest?

3.1 Three steps

Step 1 — Measure importance with LLMLingua-2 (bidirectional BERT-like model):

I_2(x_i) = P(x_i \mid \bm{x}_{\leq n}; \bm{\theta}_{\mathcal{M}_B})

A bidirectional model avoids the positional bias of causal LMs, which assign higher confidence to later tokens.

Step 2 — Prune by importance. Given target ratio $\gamma$ (e.g., 0.5), keep the top- $\gamma$ fraction of tokens:

\widetilde{\bm{c}} = \{c_i \mid I(c_i) \geq I_\gamma\}

where $I_\gamma = \text{np.percentile}([I(c_1), \dots, I(c_m)], \gamma)$

Step 3 — Train the model to generate compressed CoT. The ratio $\gamma$ is inserted after the question:

\mathcal{Q} \ [\text{EOS}] \ \gamma \ [\text{EOS}] \ \text{Compressed CoT} \ \mathcal{A}

Training mixes multiple ratios (0.5–1.0) for full-spectrum controllability.

3.2 Surprisingly cheap training

| Setting | Details | |:---|:---| | Fine-tuning | LoRA (rank=8, α=16) | | Trainable params | Only 0.2% | | Data | GSM8K 7,473 + MATH 7,500 | | 7B training time | ~2 hours (2x RTX 3090) | | 14B training time | ~2.5 hours (2x RTX 3090) |

This is not a million-dollar cluster project — it's a technique reproducible on consumer GPUs in hours.

---

4. Results: TokenSkip crushes baselines

GSM8K (7B model)

| Method | Ratio | Accuracy | Tokens | Latency | Actual ratio | |:---|:---:|:---:|:---:|:---:|:---:| | Original CoT | — | 86.2% | 213 | 5.96s | — | | Prompt 0.5 | 0.5 | 83.7% | 189 | 4.97s | 0.89 ❌ | | Truncation 0.5 | 0.5 | 7.0% 💀 | 104 | 2.95s | 0.49 ✅ | | TokenSkip 0.5 | 0.5 | 78.2% ✅ | 113 | 3.40s | 0.53 ✅ | | TokenSkip 0.7 | 0.7 | 82.5% ✅ | 150 | 4.36s | 0.70 ✅ |

Qwen2.5-14B

| Ratio | Accuracy | Tokens | Change | |:---:|:---:|:---:|:---:| | 1.0 (original) | Baseline | 313 | — | | 0.6 | Nearly equal | ~188 | < 0.4% drop | | 0.5 | -2% | ~157 | Acceptable |

Larger models appear better at finding "shortcuts" in compressed representations.

MATH-500

| Method | Ratio | Accuracy | Change | Speedup | |:---|:---:|:---:|:---:|:---:| | Original | — | 48.6% | — | 1.0x | | TokenSkip | 0.7 | 46.7% | -1.9% | 1.4x | | TokenSkip | 0.5 | 40.2% | -8.4% | 1.7x |

---

5. Most surprising: compressed CoT can be recovered

TokenSkip is not destructive compression. The model can reconstruct full reasoning from heavily compressed chains:

Compressed: "break down Deanna 26 Marcus five younger 26 - 5 21 Marcus half Leo's age twice Marcus Marcus 21, Leo's age 2 x 21 = 42"
Recovered: "Let's break it down step by step. Deanna is 26 years old. Marcus is five years younger than Deanna: M = D - 5. Marcus's age: M = 26 - 5 = 21. Marcus is half of Leo's age: M = L / 2. Leo is twice Marcus's age: L = 2M. Leo's age: L = 2 x 21 = 42."

This means you don't have to choose between inference efficiency and explainability: get answers fast with compressed CoT, then recover the full chain on demand.

---

6. Limitations and future work

1. Depends on an external importance model: LLMLingua-2 adds system complexity — could the model self-assess token importance? 2. Ratio granularity: currently global; dynamic compression that adapts during reasoning could be better. 3. Domain transfer: validated on math; effectiveness on code, scientific, and multimodal reasoning remains open. 4. Combination with training-time methods: e.g., pairing with difficulty-adaptive or RL-based approaches for greater gains.

Still, TokenSkip raises an unavoidable question: how many tokens do we actually need to think? The answer may be far fewer than we assumed.

---

Paper details

| Item | Content | |:---|:---| | Title | TokenSkip: Controllable Chain-of-Thought Compression in LLMs | | Authors | Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, Wenjie Li | | Institutions | The Hong Kong Polytechnic University, University of Science and Technology of China | | arXiv ID | 2502.12067 | | Date | 2025-02-17 | | Key results | Qwen2.5-14B GSM8K: -40% tokens, <0.4% accuracy drop; LLaMA-3.1-8B MATH-500: -30% tokens, <4% drop, 1.4x speedup | | Training cost | LoRA, 0.2% params; 7B ~2h, 14B ~2.5h (2x RTX 3090) | | Code | https://github.com/hemingkx/TokenSkip |