TokenSkip: Controllable Chain-of-Thought Compression via Selective Token Skipping
> In February 2025, a joint team from The Hong Kong Polytechnic University and the University of Science and Technology of China proposed TokenSkip, a method that analyzes token-level semantic importance within Chain-of-Thought (CoT) sequences to enable controllable compression of reasoning chains. The method achieves a 40% token reduction on Qwen2.5-14B-Instruct with less than 0.4% performance degradation, offering a third path for inference efficiency that sits apart from training-time compression and brute-force truncation.
Paper: arXiv:2502.12067 — "TokenSkip: Controllable Chain-of-Thought Compression in LLMs" by Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, Wenjie Li. Code: https://github.com/hemingkx/TokenSkip
1. Background: Redundancy in Reasoning Chains
CoT prompting significantly improves performance on complex tasks, but as models like OpenAI o1 and DeepSeek-R1 stretch CoT from hundreds to thousands or tens of thousands of tokens, inference latency and compute costs grow linearly.
| Model class | Typical CoT length | Latency impact | Bottleneck | |:---|:---|:---:|:---| | Traditional CoT | 50–200 tokens | Low | — | | o1/R1-style models | 1,000–10,000+ tokens | High | KV cache memory, quadratic attention cost |
Existing compression strategies have clear limitations:
| Category | Representative work | Mechanism | Core problem | |:---|:---|:---|:---| | Prompt constraints | Prompt-based reduction | Ask the model to shorten output | Uncontrollable ratio, inconsistent responses | | Brute truncation | Truncation | Cap max generation length | May cut critical reasoning steps | | Training-time compression | DAST, MRT, LIMR | Optimize reasoning length during training | Requires retraining | | Inference-time compression | TokenSkip | Selectively skip low-importance tokens | Needs training but is cheap |
2. Token Importance: CoT Is Not a Uniform Information Flow
Using LLMLingua-2 (Pan et al., 2024) to score token importance, the authors found:
| Token type | Examples | Importance | Function |
|:---|:---|:---:|:---|
| Math expressions | 26 - 5 = 21 | High | Core reasoning steps |
| Key entities | Deanna, Marcus | High | Problem elements |
| Numbers | 26, 21, 42 | Mid-high | Computation nodes |
| Logical connectives | so, since, therefore | Low | Semantic glue |
| Filler phrases | Let's break it down | Very low | Formatting scaffolding |
Key finding: information in CoT is highly unevenly distributed. Math formulas and entities carry the core logic; connectives and fillers mostly serve fluency.
TokenSkip uses a bidirectional (BERT-like) LM for importance scoring rather than autoregressive perplexity:
Bidirectional scoring avoids the positional bias of causal LMs, where tokens later in a sentence naturally receive higher confidence due to more context.
3. The TokenSkip Method
Given a CoT sequence \(\bm{c} = \{c_i\}_{i=1}^{m}\) and target compression ratio \(\gamma \in [0, 1]\):
1. Compute importance \(I(c_i)\) for each token 2. Set threshold: \(I_\gamma = \text{percentile}([I(c_1), \dots, I(c_m)], \gamma)\) 3. Keep tokens above the threshold: \(\widetilde{\bm{c}} = \{c_i \mid I(c_i) \geq I_\gamma\}\)
Example: "Let's break it down step by step. Deanna is 26 years old..." becomes "break down Deanna 26 Marcus five younger 26 - 5 21..." — math relations and entities survive; connectives and scaffolding are dropped.
Training setup: the model is fine-tuned on data formatted as \(\mathcal{Q} \ [\text{EOS}] \ \gamma \ [\text{EOS}] \ \text{Compressed CoT} \ \mathcal{A}\), with \(\gamma\) sampled from {0.5, 0.6, 0.7, 0.8, 0.9, 1.0}:
| Config | Setting | Benefit | |:---|:---|:---| | Fine-tuning | LoRA (rank=8, α=16) | Only ~0.2% of parameters trained | | Data | GSM8K 7,473 + MATH 7,500 | Small, high-quality | | Time | 7B: ~2h, 14B: ~2.5h | Feasible on 2× RTX 3090 | | Original CoT retained | Mixed γ=1.0 samples | Prevents over-compression degradation |
At inference, users control the trade-off via γ: 1.0 = no compression, 0.7 = balanced, 0.5 = maximum efficiency.
4. Evaluation
GSM8K (Qwen2.5-14B-Instruct)
| Method | Target ratio | Actual ratio | Accuracy | Latency | |:---|:---:|:---:|:---:|:---:| | Original CoT | — | — | 86.2% | 5.96s | | Prompt 0.5 | 0.5 | 0.89 ❌ | 83.7% | 4.97s | | Truncation 0.5 | 0.5 | 0.49 ✅ | 7.0% 💀 | 2.95s | | TokenSkip 0.5 | 0.5 | 0.53 ✅ | 78.2% ✅ | 3.40s | | TokenSkip 0.7 | 0.7 | 0.70 ✅ | 82.5% ✅ | 4.36s |
Prompt-based reduction badly misses its target ratio (0.89 vs 0.5), and truncation at 0.5 collapses accuracy from 86.2% to 7.0% — blind truncation destroys reasoning. TokenSkip hits its ratio accurately while preserving performance.
Model scale
| Model | Ratio | Accuracy change | Observation | |:---|:---:|:---:|:---| | 7B | 0.6 | -5.1% | Smaller models more compression-sensitive | | 14B | 0.6 | < -0.4% | Larger models nearly unaffected | | 14B | 0.5 | -2.0% | Robust even at high compression |
MATH-500 (LLaMA-3.1-8B)
| Setting | Ratio | Accuracy | Change | Speedup | |:---|:---:|:---:|:---:|:---:| | Original | — | 48.6% | — | 1.0x | | TokenSkip | 0.7 | 46.7% | -1.9% | 1.4x | | TokenSkip | 0.5 | 40.2% | -8.4% | 1.7x |
5. Recoverability of Compressed CoT
A unique property of TokenSkip is non-destructive compression: the model can reconstruct full reasoning chains from compressed ones. For example, the compressed "break down Deanna 26 Marcus five younger 26 - 5 21 Marcus half Leo's age twice Marcus Marcus 21, Leo's age 2 x 21 = 42" can be restored to a complete step-by-step derivation. In deployment, one can serve the compressed version for fast answers and restore the full chain when auditability or explanation is needed.
6. Positioning and Open Questions
| Path | Methods | Intervention | Idea | Complexity | |:---|:---|:---:|:---|:---:| | Training-time optimization | DAST, MRT, LIMR | Training | Change reasoning behavior | High | | Post-hoc compression | TokenSkip | Post-training fine-tune | Teach selective skipping | Medium | | Architecture/decoding | Speculative Decoding, Medusa | Inference | Accelerate decoding | Medium |
TokenSkip's advantages are low cost and controllability: no retraining pipeline redesign, no architecture changes — just lightweight LoRA fine-tuning.
Open questions:
1. Dynamic compression — adjust the ratio during reasoning rather than using a global γ. 2. Self-assessed importance — can the model learn to skip tokens without an external bidirectional LM? 3. Cross-domain generalization — does the math-reasoning success transfer to code generation, scientific reasoning, or multimodal reasoning? 4. Synergy with training-time methods — e.g., combining with DAST's difficulty-adaptive budgets.
7. Conclusion
TokenSkip demonstrates that CoT sequences contain substantial compressible redundancy. Its core methodology — importance-based selective skipping — delivers meaningful efficiency gains at minimal cost, without retraining or architectural changes: simply teaching the model to "think with skips."
Paper Details
| Item | Content | |:---|:---| | Title | TokenSkip: Controllable Chain-of-Thought Compression in LLMs | | Authors | Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, Wenjie Li | | Institutions | The Hong Kong Polytechnic University, University of Science and Technology of China | | arXiv | 2502.12067 | | Date | 2025-02-17 | | Key results | Qwen2.5-14B GSM8K: -40% tokens, <0.4% accuracy drop; LLaMA-3.1-8B MATH-500: -30% tokens, 1.4x speedup | | Training | LoRA (rank=8, α=16), ~0.2% params, 2x RTX 3090, 2–2.5 hours | | Code | https://github.com/hemingkx/TokenSkip |