TokenSkip: Controllable Chain-of-Thought Compression for LLMs — Method and Insights

Forum topic · 小凯 · 2026-05-11

Summary

TokenSkip, proposed in February 2025 by researchers from The Hong Kong Polytechnic University and the University of Science and Technology of China (arXiv:2502.12067), is a controllable method for compressing Chain-of-Thought (CoT) reasoning in large language models. The approach analyzes token-level semantic importance within CoT sequences using a bidirectional language model (LLMLingua-2), then trains the model with lightweight LoRA fine-tuning to selectively skip low-importance tokens such as filler phrases and connectives while preserving mathematical expressions, numbers, and key entities. A compression ratio gamma specified at inference time gives users direct control over the efficiency-accuracy trade-off. Experiments show that on GSM8K, Qwen2.5-14B-Instruct achieves a 40% token reduction with performance degradation below 0.4%, while on MATH-500, LLaMA-3.1-8B gains a 1.4x speedup at a 30% compression ratio with only 1.9% accuracy loss. Compared with prompt-based reduction (uncontrollable) and truncation (catastrophic accuracy loss), TokenSkip offers accurate compression ratios and strong performance retention. Notably, compressed CoT remains recoverable to full reasoning chains, supporting auditability. Training requires only ~2-2.5 hours on two RTX 3090 GPUs. Code is available at https://github.com/hemingkx/TokenSkip.

TokenSkip: When Reasoning Chains Meet Selective Forgetting — Methodology and Insights Behind Controllable CoT Compression

> In February 2025, a joint team from The Hong Kong Polytechnic University and the University of Science and Technology of China proposed TokenSkip, a method that achieves controllable Chain-of-Thought (CoT) compression by analyzing token-level semantic importance in CoT sequences. It delivers a 40% token reduction on Qwen2.5-14B-Instruct with performance loss below 0.4%, offering a third path for inference efficiency distinct from training-time compression and brute truncation.

---

1. Background: Redundancy in Reasoning Chains

CoT prompting significantly improves performance on complex tasks, but as models like OpenAI o1 and DeepSeek-R1 extend CoT from hundreds to tens of thousands of tokens, inference latency and compute cost grow linearly.

| Model type | Typical CoT length | Latency impact | Bottleneck | |:---:|:---:|:---:|:---| | Traditional CoT | 50–200 tokens | Low | — | | o1/R1-class models | 1,000–10,000+ tokens | High | KV cache memory, quadratic attention cost |

Core tension: longer CoT may improve reasoning, but linear latency growth makes long CoT expensive in deployment.

Existing compression strategies and their limits:

| Category | Representative | Mechanism | Key problem | |:---|:---|:---|:---| | Prompt constraints | Prompt-based reduction | Ask the model to shorten output | Compression ratio uncontrollable, inconsistent responses | | Brute truncation | Truncation | Cap max generation length | May cut critical reasoning steps | | Training-time compression | DAST, MRT, LIMR | Optimize reasoning length during training | Requires retraining | | Inference-time compression | TokenSkip | Selectively skip low-importance tokens | Needs light training but is cheap |

---

2. Token Importance: CoT Is Not a Uniform Information Stream

Using LLMLingua-2 (Pan et al., 2024), the authors found a highly uneven information distribution:

| Token category | Examples | Importance | Function | |:---|:---|:---:|:---| | Math expressions | 26 - 5 = 21 | High | Core reasoning steps | | Key entities | Deanna, Marcus | High | Problem element identification | | Numbers | 26, 21, 42 | Mid-high | Computation nodes | | Logical connectives | so, since, therefore | Low | Semantic glue | | Fillers | Let's break it down | Very low | Formatting scaffolding |

TokenSkip uses a bidirectional (BERT-like) LM rather than autoregressive perplexity to score importance:

I_2(x_i) = P(x_i \mid \bm{x}_{\leq n}; \bm{\theta}_{\mathcal{M}_B})

Causal LMs suffer positional bias — tokens later in a sentence naturally have higher confidence due to more context. Bidirectional models remove this bias and score tokens more fairly.

---

3. The TokenSkip Method: Selective Skipping and Controllable Compression

Given a CoT sequence $\bm{c} = \{c_i\}_{i=1}^{m}$ and a target compression ratio $\gamma \in [0, 1]$ :

1. Compute each token's importance $$I(c_i)$$ 2. Determine the threshold: $I_\gamma = \text{percentile}([I(c_1), \dots, I(c_m)], \gamma)$ 3. Keep tokens above the threshold: $\widetilde{\bm{c}} = \{c_i \mid I(c_i) \geq I_\gamma\}$

Example: "Let's break it down step by step. Deanna is 26 years old..." becomes "break down Deanna 26 Marcus five younger 26 - 5 21..." — all math relations and entities preserved, connectives and scaffolding removed.

Training paradigm

Training data format:

\mathcal{Q} \ [\text{EOS}] \ \gamma \ [\text{EOS}] \ \text{Compressed CoT} \ \mathcal{A}

with $\gamma$ sampled from $\{0.5, 0.6, 0.7, 0.8, 0.9, 1.0\}$ so the model learns across the full spectrum of compression levels.

| Setting | Value | Advantage | |:---|:---|:---| | Fine-tuning | LoRA (rank=8, α=16) | Only ~0.2% of parameters trained | | Data | GSM8K 7,473 + MATH 7,500 | Small scale, high quality | | Training time | 7B: ~2h, 14B: ~2.5h | Feasible on 2× RTX 3090 | | Original CoT mixed in | γ=1.0 samples | Prevents over-compression degradation |

Controllability at inference

Users specify $\gamma$ at inference: γ=1.0 gives the original CoT, γ=0.7 balances efficiency and accuracy, γ=0.5 maximizes efficiency. Interactive apps may prefer high compression (low latency); critical decision scenarios may prefer low compression (high accuracy).

---

4. Evaluation

GSM8K performance (Qwen2.5-14B-Instruct)

| Method | Target ratio | Actual ratio | Accuracy | Tokens | Latency | |:---|:---:|:---:|:---:|:---:|:---:| | Original CoT | — | — | 86.2% | 213 | 5.96s | | Prompt 0.5 | 0.5 | 0.89 ❌ | 83.7% | 189 | 4.97s | | Truncation 0.5 | 0.5 | 0.49 ✅ | 7.0% 💀 | 104 | 2.95s | | TokenSkip 0.5 | 0.5 | 0.53 ✅ | 78.2% ✅ | 113 | 3.40s | | TokenSkip 0.7 | 0.7 | 0.70 ✅ | 82.5% ✅ | 150 | 4.36s |

Prompt-based reduction badly misses its target (0.89 actual vs. 0.5 requested); truncation hits the ratio but collapses accuracy from 86% to 7%, confirming that blind truncation destroys reasoning. TokenSkip best balances accurate ratios and performance retention.

Model scaling (GSM8K, Qwen2.5-Instruct)

| Model | Ratio | Accuracy change | Observation | |:---|:---:|:---:|:---| | 7B | 0.6 | -5.1% | Smaller models are more compression-sensitive | | 14B | 0.6 | < -0.4% | Large models nearly unaffected | | 14B | 0.5 | -2.0% | Robust even at high compression |

Larger models handle compressed (sparse) reasoning representations more robustly.

MATH-500 (LLaMA-3.1-8B)

| Method | Ratio | Accuracy | Change | Speedup | |:---|:---:|:---:|:---:|:---:| | Original | — | 48.6% | — | 1.0x | | TokenSkip | 0.7 | 46.7% | -1.9% | 1.4x | | TokenSkip | 0.5 | 40.2% | -8.4% | 1.7x |

---

5. Recoverability of Compressed CoT

A unique property of TokenSkip is non-destructive compression: the model can reconstruct the full reasoning chain from the compressed version.

| Version | Content | |:---|:---| | Compressed | "break down Deanna 26 Marcus five younger 26 - 5 21 Marcus half Leo's age twice Marcus Marcus 21, Leo's age 2 x 21 = 42" | | Recovered | "Let's break it down step by step. Deanna is 26 years old. Marcus is five years younger than Deanna: M = D - 5. Marcus's age: M = 26 - 5 = 21..." |

In deployment, one can answer quickly with the compressed CoT and recover the full version later for auditing or explanation.

---

6. Discussion: Where TokenSkip Fits

| Path | Methods | Intervention point | Idea | Complexity | |:---|:---|:---:|:---|:---:| | Training-time optimization | DAST, MRT, LIMR | Training | Change reasoning behavior | High | | Post-training compression | TokenSkip | Post-training LoRA | Teach selective token skipping | Medium | | Architecture/decoding | Speculative Decoding, Medusa | Inference | Accelerate decoding | Medium |

TokenSkip's advantage is low cost and controllability — no retraining pipeline redesign or architecture change, just lightweight LoRA fine-tuning.

Open questions

1. Dynamic compression: adjust γ during reasoning — keep more tokens at critical steps, compress more aggressively elsewhere? 2. Self-assessed importance: can the model itself learn to score and skip tokens instead of relying on an external bidirectional LM (LLMLingua-2)? 3. Cross-domain generalization: does the math-reasoning success transfer to code generation, scientific reasoning, or multimodal reasoning? 4. Synergy with training-time methods: could TokenSkip combine with DAST's difficulty-adaptive budgets — allocate budget adaptively, then skip selectively within it?

---

7. Conclusion

TokenSkip demonstrates that CoT sequences contain significant compressible redundancy. Its core methodology — importance-based selective skipping — yields substantial efficiency gains at minimal implementation cost: no model retraining, no architecture change, just teaching the model to "think in jumps."

---

Paper details

| Item | Content | |:---|:---| | Title | TokenSkip: Controllable Chain-of-Thought Compression in LLMs | | Authors | Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, Wenjie Li | | Affiliations | The Hong Kong Polytechnic University, University of Science and Technology of China | | arXiv ID | 2502.12067 | | Date | 2025-02-17 | | Key results | Qwen2.5-14B GSM8K: -40% tokens, < 0.4% performance drop; LLaMA-3.1-8B MATH-500: -30% tokens, 1.4x speedup | | Training config | LoRA (rank=8, α=16), ~0.2% parameters; 2x RTX 3090; 2–2.5 hours | | Code | https://github.com/hemingkx/TokenSkip |