ExpThink: Experience-Guided RL Framework for Adaptive Chain-of-Thought Compression

Forum topic · 小凯 · 2026-05-11

Summary

ExpThink is a reinforcement learning framework introduced by Bian et al. (arXiv:2605.07501) that adaptively compresses Chain-of-Thought (CoT) reasoning without sacrificing accuracy. It addresses two structural flaws of conventional length-penalty RL approaches—uniform penalties that ignore problem difficulty and static coefficients that fail to track model improvement—through dual mechanisms. First, experience-guided reward shaping tracks the shortest correct solution (personal best) per question and applies a three-tier reward, creating a self-evolving curriculum whose compression standard tightens automatically as the model grows stronger. Second, difficulty-adaptive advantage estimation replaces standard-deviation normalization with correct-count normalization, amplifying gradients on hard questions (preserving accuracy) while suppressing them on easy ones (encouraging brevity). Across multiple mathematical reasoning benchmarks, ExpThink reduces average response length by up to 77% while improving accuracy, achieving a 3x accuracy-efficiency ratio and outperforming fixed-penalty RL and TokenSkip-style baselines. The method is plug-and-play and can be layered onto existing RL algorithms, marking a shift from externally penalized compression to intrinsically motivated concise reasoning.

ExpThink: Experience-Guided RL Framework for Adaptive Chain-of-Thought Compression

> In May 2026, Bian et al. proposed ExpThink, an RL framework for adaptive Chain-of-Thought (CoT) compression built on dual mechanisms. Targeting the limitations of uniform static length penalties in existing methods, it introduces experience-guided reward shaping—which tracks each question's shortest correct solution and applies a dynamic three-tier reward to form a self-evolving curriculum—and difficulty-adaptive advantage, which replaces standard-deviation normalization with correct-count normalization for monotonic difficulty-scaled gradient allocation. On multiple mathematical reasoning benchmarks, ExpThink cuts average response length by up to 77% while improving accuracy, achieving up to a 3x accuracy-efficiency ratio.

1. Background: Limitations of RL-Based CoT Compression

Existing RL-based CoT compression typically applies a uniform length penalty:

R = R_{\text{correct}} - \lambda \cdot \text{length}

This design has three structural flaws:

| Flaw | Manifestation | Consequence | |:---|:---|:---| | Uniformity | Same penalty coefficient for easy and hard questions | Over-compression of easy problems, under-compression of hard ones | | Staticness | $\lambda$ fixed before training | Standard not updated as model capability improves | | No memory | Historical best solutions unused | Repeated exploration of already-known feasible solution space |

Two dimensions require adaptivity: model capability dynamics (the model grows stronger during training) and problem difficulty differences (intrinsic per-question difficulty). Standard methods handle neither.

2. Method: Dual Mechanisms

2.1 Experience-Guided Reward Shaping

A dynamic personal best is maintained per question $$q$$ :

B_q = \min\{\text{length}(r) : r \text{ is correct for } q\}

The three-tier reward function:

| Response | Condition | Reward | |:---|:---|:---:| | Correct + record-breaking brevity | $\text{length} < B_q$ | $R_{\max}$ | | Correct + verbose | $\text{length} \geq B_q$ | $R_{\max} \cdot \frac{B_q}{\text{length}}$ | | Incorrect | — | 0 |

This creates a self-evolving curriculum: as the model's personal bests shrink over training (e.g., 100 → 50 → 20 tokens), the effective compression standard tightens automatically without manual scheduling.

2.2 Difficulty-Adaptive Advantage

Standard advantage normalization,

A_i = \frac{R_i - \bar{R}}{\sigma_R}

shares the same statistics across all questions, ignoring difficulty. ExpThink uses correct-count normalization:

A_i^{\text{adaptive}} = \frac{R_i - \bar{R}_d}{C_d}

where $$d$$ is the difficulty level and $$C_d$$ is the cumulative correct count at that level:

Easy questions (high $$C_d$$ ): gradients suppressed → model focuses on compression
Hard questions (low $$C_d$$ ): gradients amplified → model preserves accuracy

Since $$C_d$$ is monotonically related to difficulty, gradient scaling is provably difficulty-correlated.

3. Experimental Results

| Metric | Baseline | ExpThink | Change | |:---|:---:|:---:|:---:| | Average response length | 100% | 23% | -77% | | Accuracy | baseline | improved | positive | | Accuracy-efficiency ratio | 1x | 3x | +200% |

Compared against fixed-penalty RL (accuracy drop, 1.5x efficiency) and TokenSkip (accuracy maintained, 2x efficiency), ExpThink achieves the highest compression rate while *improving* accuracy, with consistent results across multiple mathematical reasoning benchmarks.

4. Theoretical Analysis

The shaped reward creates a natural race incentive:

\frac{\partial R}{\partial \text{length}} = -\frac{R_{\max} \cdot B_q}{\text{length}^2}

Longer responses yield smaller marginal rewards, inherently encouraging compression. Correct-count normalization separates optimization goals: accuracy dominates on hard questions, compression on easy ones.

5. Relation to Related Work

TokenSkip: post-hoc CoT compression; ExpThink internalizes compression during training, generating shorter reasoning from the source.
DAST: difficulty-aware compute allocation at inference; ExpThink adjusts gradients by difficulty at training—an orthogonal dimension.
80/20 Rule: ExpThink trains models to focus on critical tokens automatically.
Rubric-Grounded RL: ExpThink's three-tier reward is a partial-credit idea applied to compression.
Coupling Tax: ExpThink prevents length-induced answer-space crowding at the training level.

6. Limitations and Future Directions

Personal-best initialization for new questions (e.g., difficulty-level statistics, progressive tightening)
Explicit difficulty models (supervised prediction, online estimation) beyond implicit correct counts
Cross-domain transfer to code generation, scientific reasoning, and creative tasks
Combination with inference-time optimization (e.g., dynamic routing, evaluation deduplication)

7. Conclusion

ExpThink provides an adaptive, self-evolving RL framework for CoT compression through experience-guided reward shaping and difficulty-adaptive advantage. Core contributions: dynamic standards via personal-best tracking, difficulty-aware gradient allocation via correct-count normalization, dual wins of -77% length and improved accuracy, and plug-and-play compatibility with existing RL algorithms—representing a paradigm shift from externally penalized to intrinsically motivated compression.

Paper Details

| Item | Content | |:---|:---| | Title | ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression | | Authors | Tingcheng Bian, Yuzhe Zhang, Jing Jin, Jinchang Luo, MingQuan Cheng, Haiwei Wang, Wenyuan Jiang, Miaohui Wang | | arXiv ID | 2605.07501 | | Date | 2026-05-08 | | Key results | Response length -77%; accuracy improved; 3x accuracy-efficiency ratio; outperforms existing RL compression methods |

ExpThink: Experience-Guided RL Framework for Adaptive Chain-of-Thought Compression

Summary

ExpThink: Experience-Guided RL Framework for Adaptive Chain-of-Thought Compression

1. Background: Limitations of RL-Based CoT Compression

2. Method: Dual Mechanisms

2.1 Experience-Guided Reward Shaping

2.2 Difficulty-Adaptive Advantage

3. Experimental Results

4. Theoretical Analysis

5. Relation to Related Work

6. Limitations and Future Directions

7. Conclusion

Paper Details

Tags