TokenSkip: Controllable Chain-of-Thought Compression for LLM Reasoning Efficiency
> In February 2025, a joint team from The Hong Kong Polytechnic University and the University of Science and Technology of China proposed TokenSkip, a method for controllable Chain-of-Thought (CoT) compression based on token-level semantic importance analysis. It achieves a 40% token reduction on Qwen2.5-14B-Instruct with less than 0.4% performance drop, offering a third path between training-time compression and blunt truncation.
1. Background: The Redundancy of Reasoning Chains
CoT prompting boosts performance on complex tasks, but as models like OpenAI o1 and DeepSeek-R1 extend reasoning to thousands of tokens, latency and compute costs grow linearly.
| Model type | Typical CoT length | Latency impact | Bottleneck | |:---|:---|:---|:---| | Traditional CoT | 50–200 tokens | Low | — | | o1/R1-class models | 1,000–10,000+ tokens | High | KV cache memory, quadratic attention cost |
Existing approaches have clear limits:
- Prompt-based reduction: compression ratio is uncontrollable and model response is inconsistent.
- Truncation: may cut off critical reasoning steps.
- Training-time compression (DAST, MRT, LIMR): requires retraining the model.
- TokenSkip (inference-time compression): selectively skips low-importance tokens; requires light additional training.
2. Token Importance: CoT Is Not a Uniform Information Stream
Using LLMLingua-2 (Pan et al., 2024) to score tokens, the authors found:
| Token category | Examples | Importance |
|:---|:---|:---|
| Math expressions | 26 - 5 = 21 | High |
| Key entities | Deanna, Marcus | High |
| Numbers | 26, 21, 42 | Medium-high |
| Logical connectives | so, since, therefore | Low |
| Filler phrases | Let's break it down | Very low |
Importance is measured with a bidirectional (BERT-like) model rather than autoregressive perplexity, because causal attention has a position bias—tokens near the end of a sentence naturally receive higher confidence. The bidirectional formulation:
3. The TokenSkip Method
Given a CoT sequence and target compression ratio \(\gamma\), TokenSkip:
1. Computes each token's importance \(I(c_i)\). 2. Sets a percentile threshold \(I_\gamma = \text{percentile}([I(c_1), \dots, I(c_m)], \gamma)\). 3. Keeps tokens above the threshold: \(\widetilde{\bm{c}} = \{c_i \mid I(c_i) \geq I_\gamma\}\).
Example: "Let's break it down step by step. Deanna is 26 years old..." becomes "break down Deanna 26 Marcus five younger 26 - 5 21..." — all math relations and entities preserved, connectives and scaffolding removed.
Training format mixes multiple ratios (\(\gamma \in \{0.5, ..., 1.0\}\)):
| Configuration | Setting | |:---|:---| | Fine-tuning | LoRA (rank=8, α=16), only 0.2% of parameters | | Data | GSM8K 7,473 + MATH 7,500 samples | | Time | ~2h for 7B, ~2.5h for 14B on 2× RTX 3090 | | Safeguard | γ=1.0 samples mixed in to prevent capability loss |
At inference, users specify γ (1.0 = no compression, 0.7 = balanced, 0.5 = aggressive).
4. Experimental Results
GSM8K (Qwen2.5-14B-Instruct)
| Method | Actual compression | Accuracy | Latency | |:---|:---|:---|:---| | Original CoT | — | 86.2% | 5.96s | | Prompt 0.5 | 0.89 (missed target) | 83.7% | 4.97s | | Truncation 0.5 | 0.49 | 7.0% | 2.95s | | TokenSkip 0.5 | 0.53 | 78.2% | 3.40s | | TokenSkip 0.7 | 0.70 | 82.5% | 4.36s |
Key takeaways: prompt-based reduction badly misses its target ratio (0.89 vs. 0.5), and truncation at 0.5 collapses accuracy from 86% to 7%, confirming that blind truncation destroys reasoning. TokenSkip hits its target ratios accurately while retaining most performance.
Model scale
| Model | Compression | Accuracy change | |:---|:---|:---| | 7B | 0.6 | -5.1% | | 14B | 0.6 | < -0.4% | | 14B | 0.5 | -2.0% |
Larger models are far more robust to compression.
MATH-500 (LLaMA-3.1-8B)
| Setting | Accuracy | Speedup | |:---|:---|:---| | Original | 48.6% | 1.0x | | TokenSkip 0.7 | 46.7% (-1.9%) | 1.4x | | TokenSkip 0.5 | 40.2% (-8.4%) | 1.7x |
5. Recoverability: Compression Without Losing Explainability
TokenSkip compression is non-destructive: the fine-tuned model can reconstruct the full reasoning chain from a compressed one (e.g., recovering "Deanna is 26 years old. Marcus is five years younger... M = 26 - 5 = 21..." from the pruned token stream). In deployment, this enables fast answers from compressed CoT with full auditability on demand.
6. Positioning and Open Questions
TokenSkip sits between training-time optimization (DAST, MRT, LIMR) and architecture/decoding-level speedups (speculative decoding, Medusa). Its advantages are a low barrier to adoption and explicit controllability.
Open questions:
1. Dynamic compression — adjusting the ratio per step instead of globally. 2. Self-assessed importance — can the model learn to skip tokens without an external scorer? 3. Cross-domain generalization — code generation, scientific reasoning, multimodal CoT. 4. Synergy with training-time methods — combining difficulty-adaptive budgets with selective skipping.
7. Conclusion
TokenSkip demonstrates that reasoning chains contain substantial compressible redundancy. By teaching a model to "skip while thinking" via lightweight LoRA fine-tuning, it delivers significant efficiency gains without retraining from scratch or architectural changes.
Paper Details
| Item | Content | |:---|:---| | Title | TokenSkip: Controllable Chain-of-Thought Compression in LLMs | | Authors | Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, Wenjie Li | | Institutions | The Hong Kong Polytechnic University; University of Science and Technology of China | | arXiv ID | 2502.12067 | | Date | 2025-02-17 | | Key results | Qwen2.5-14B GSM8K: -40% tokens, < 0.4% accuracy drop; LLaMA-3.1-8B MATH-500: -30% tokens, 1.4x speedup | | Code | https://github.com/hemingkx/TokenSkip |