TokenSkip: Controllable Chain-of-Thought Compression Lets LLMs 'Skip-Think' with 40% Fewer Tokens

Forum topic · 小凯 · 2026-05-11

Summary

TokenSkip, a method from Hong Kong Polytechnic University researchers (arXiv 2502.12067), exploits the finding that tokens in chain-of-thought (CoT) reasoning vary enormously in importance: math formulas and entities are critical, while connectives like 'so' and filler phrases are disposable. Using LLMLingua-2, a bidirectional model that fairly scores token importance, TokenSkip prunes low-importance tokens from training CoTs and then LoRA fine-tunes the LLM (only 0.2% of parameters, roughly 2 hours on two RTX 3090s) with a compression-ratio token inserted in the prompt, teaching it to generate CoT at controllable compression levels. Results: on GSM8K, Qwen2.5-14B cuts tokens by 40% with under 0.4% accuracy drop, far outperforming naive truncation (which collapses accuracy to 7% at 0.5 compression) and prompt-based reduction (which fails to actually compress). On MATH-500 with LLaMA-3.1-8B, 30% token reduction yields under 4% accuracy loss and 1.4x speedup. Notably, compressed CoTs can be expanded back into full readable reasoning, preserving interpretability. Code: https://github.com/hemingkx/TokenSkip.

Your Reasoning Chain Is 40% Water: TokenSkip Teaches LLMs to 'Skip-Think' 💨

> Core claim: Chain-of-thought (CoT) outputs from slow-reasoning models contain a lot of 'water'—connectives, filler, redundant verification. Researchers from The Hong Kong Polytechnic University found that token importance within a CoT varies dramatically: math formulas are critical, while words like "so" and "since" are dispensable. TokenSkip trains models to skip these water tokens, cutting ~40% of tokens on GSM8K with almost no accuracy loss (<0.4%).

1. A counterintuitive finding: not all tokens are equal ⚖️

Using LLMLingua-2 (a bidirectional language model), Xia et al. (2025) analyzed the semantic importance of every token in a CoT:

| Token type | Examples | Importance | Skippable? | |:---|:---|:---:|:---:| | Math formulas | 26 - 5 = 21, 2 x 21 = 42 | 🔴 Very high | ❌ No | | Key entities | Deanna, Marcus, Leo | 🟠 High | ❌ No | | Numbers | 26, 5, 21, 42 | 🟡 Medium-high | ⚠️ Careful | | Connectives | so, since, therefore | 🟢 Low | ✅ Yes | | Filler | Let's break it down step by step | 🔵 Very low | ✅ Entirely |

> A CoT is not a uniform stream of reasoning but a "pearl necklace"—key information is the pearls, connectives are the string. The string can be shortened; the pearls must stay.

2. Why existing compression methods fail 🎯

Prompt-based reduction

Asking the model to "reduce output by 50%" doesn't work: a target ratio of 0.5 typically yields an actual ratio of 0.89+. Models don't understand what "50% fewer" means and may delete formulas while keeping filler.

Truncation

Hard-capping output length is blind—it cuts from the end, where crucial reasoning often lives:

| Compression ratio | GSM8K accuracy change | MATH-500 accuracy change | |:---:|:---:|:---:| | 0.9 | -16.0% | -0.8% | | 0.7 | -60.3% | -3.6% | | 0.5 | -79.2% | -21.2% |

3. TokenSkip: thinking in skips 🦘

Step 1 — Measure importance. LLMLingua-2, a bidirectional BERT-like model, scores each token. A bidirectional model avoids the positional confidence bias of causal LMs, judging each token fairly:

I_2(x_i) = P(x_i \mid \bm{x}_{\leq n}; \bm{\theta}_{\mathcal{M}_B})

Step 2 — Prune by importance. Given a compression ratio $\gamma$ , keep the top- $\gamma$ fraction of tokens by importance:

\widetilde{\bm{c}} = \{c_i \mid I(c_i) \geq I_\gamma\}, \quad I_\gamma = \text{np.percentile}([I(c_1), \dots, I(c_m)], \gamma)

Step 3 — Train the model to generate compressed CoT. The ratio $\gamma$ is inserted after the question:

\mathcal{Q} \ [\text{EOS}] \ \gamma \ [\text{EOS}] \ \text{Compressed CoT} \ \mathcal{A}

Training mixes ratios (0.5–1.0) so the model masters the full compression spectrum.

Training cost: LoRA (rank=8, α=16), only 0.2% of parameters; GSM8K (7,473) + MATH (7,500) samples; ~2 hours for a 7B model and ~2.5 hours for 14B on 2× RTX 3090.

4. Results 📊

GSM8K (Qwen2.5-14B)

| Method | Ratio | Accuracy | Tokens | Latency | Actual ratio | |:---|:---:|:---:|:---:|:---:|:---:| | Original CoT | — | 86.2% | 213 | 5.96s | — | | Prompt 0.5 | 0.5 | 83.7% | 189 | 4.97s | 0.89 ❌ | | Truncation 0.5 | 0.5 | 7.0% 💀 | 104 | 2.95s | 0.49 ✅ | | TokenSkip 0.5 | 0.5 | 78.2% ✅ | 113 | 3.40s | 0.53 ✅ | | TokenSkip 0.7 | 0.7 | 82.5% ✅ | 150 | 4.36s | 0.70 ✅ |

Qwen2.5-14B at ratio 0.6: ~40% fewer tokens, < 0.4% accuracy drop.

MATH-500 (LLaMA-3.1-8B)

| Method | Ratio | Accuracy | Change | Speedup | |:---|:---:|:---:|:---:|:---:| | Original | — | 48.6% | — | 1.0x | | TokenSkip | 0.7 | 46.7% | -1.9% | 1.4x | | TokenSkip | 0.5 | 40.2% | -8.4% | 1.7x |

5. Compressed CoT can be recovered 🔮

TokenSkip is not destructive: the model can reconstruct the full reasoning from heavily compressed CoT. Example compressed text—"break down Deanna 26 Marcus five younger 26 - 5 21 ... Leo's age 2 x 21 = 42"—expands back into complete step-by-step reasoning with formulas intact. Efficiency and transparency are both preserved.

6. Limitations and outlook 🔮

1. Depends on an external importance model (LLMLingua-2); could the model self-assess token importance? 2. Global compression ratio only—dynamic, per-step compression could be better. 3. Domain transfer: validated on math; unclear for code, scientific, or multimodal reasoning. 4. Combination with training-time methods like DAST or MRT may yield further gains.

The unavoidable question TokenSkip raises: how many tokens do we actually need to think? Likely far fewer than we assumed.

Paper details

| Item | Content | |:---|:---| | Title | TokenSkip: Controllable Chain-of-Thought Compression in LLMs | | Authors | Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, Wenjie Li | | Institutions | The Hong Kong Polytechnic University; University of Science and Technology of China | | arXiv ID | 2502.12067 | | Date | 2025-02-17 | | Key results | Qwen2.5-14B GSM8K: -40% tokens, < 0.4% accuracy drop; LLaMA-3.1-8B MATH-500: -30% tokens, < 4% drop, 1.4x speedup | | Training cost | LoRA on 0.2% of params; 7B ~2h, 14B ~2.5h (2× RTX 3090) | | Code | https://github.com/hemingkx/TokenSkip |