ExpThink: Experience-Guided RL for Chain-of-Thought Compression Cuts Response Length by 77% While Boosting Accuracy

Forum topic · 小凯 · 2026-05-11

Summary

ExpThink (Bian et al., 2026) introduces a reinforcement learning framework for chain-of-thought (CoT) compression that combines two mechanisms: experience-guided reward shaping and difficulty-adaptive advantage. Instead of a fixed length penalty, the method tracks a per-question personal best (shortest correct solution) as a dynamic standard, granting full reward only when a correct answer is shorter than this record, discounted reward when longer, and zero for incorrect answers; the standard tightens automatically as the model improves. Difficulty-adaptive advantage normalizes by correct-count per difficulty level, amplifying gradients on hard problems and suppressing them on easy ones. Reported results show response length reduced by up to 77%, accuracy improved rather than degraded, and a 3x accuracy-efficiency ratio versus baselines and prior RL compression methods like fixed length penalties. The approach is simple to implement, compatible with GRPO, PPO, and REINFORCE++, and requires only storing best-solution lengths per question.

ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression

A forum post on zhichai.net reviews a 2026 paper (arXiv: 2605.07501) proposing ExpThink, a reinforcement learning approach to chain-of-thought (CoT) compression. Its headline claim: response length cut by up to 77% while accuracy *improves*, yielding a 3× accuracy-efficiency ratio.

Key points

Problem with existing methods

Standard RL-based CoT compression applies a fixed penalty:

R = R_{\text{correct}} - \lambda \cdot \text{length}

Criticisms raised in the post:

Uniform penalty: easy and hard problems are penalized identically, over-penalizing the former and under-penalizing the latter.
Static standard: λ is fixed before training and never adapts as the model improves.
No memory of history: the model never knows it previously solved a question in 50 tokens.

Engine 1: Experience-guided reward shaping

Each question keeps a personal best record (shortest correct solution so far):

| Outcome | Reward | |:---|:---:| | Correct and shorter than personal best | Full reward | | Correct but longer | Discounted (larger discounts for longer responses) | | Incorrect | Zero |

This creates a self-evolving curriculum: early in training, personal bests are long (easy full rewards); as the model improves, bests shrink and the standard automatically tightens — no manual schedule needed.

Engine 2: Difficulty-adaptive advantage

Standard RL normalizes advantages across all problems:

A_i = \frac{R_i - \bar{R}}{\sigma_R}

ExpThink instead normalizes by correct count per difficulty level:

A_i^{\text{adaptive}} = \frac{R_i - \bar{R}_d}{\text{correct\_count}_d}

Easy problems (high correct count): gradients suppressed — reward brevity.
Hard problems (low correct count): gradients amplified — accuracy matters most.

Reported results

| Metric | Baseline | ExpThink | Change | |:---|:---:|:---:|:---:| | Avg. response length | 100% | 23% | -77% | | Accuracy | baseline | improved | positive | | Accuracy-efficiency ratio | baseline | 3× | +200% |

The post claims ExpThink outperforms fixed-length-penalty RL and TokenSkip on both compression and accuracy, with consistent results across multiple math reasoning benchmarks.

Connections to prior work

TokenSkip: compresses CoT statically; ExpThink trains brevity in from the start.
DAST: difficulty-based compute allocation, realized here at the reward-design level.
80/20 rule: the model learns to focus on the ~20% high-entropy tokens.
Rubric-Grounded RL: the three-tier reward is a partial-credit scheme.
Coupling Tax: compact reasoning chains free answer space.

Author's outlook

The forum author predicts experience-guided reward shaping will become standard in RL-based CoT compression by end of 2026, citing simple implementation (a dictionary of personal bests, a normalization statistic), compatibility with GRPO/PPO/REINFORCE++, and no manual tuning.

Open questions

How to initialize personal bests for new questions.
How exactly to define "difficulty."
Whether personal bests should update across multi-turn interactions.
Generalization beyond math to code, science, and creative writing.

Paper details

| Item | Detail | |:---|:---| | Title | ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression | | Authors | Tingcheng Bian, Yuzhe Zhang, Jing Jin, Jinchang Luo, MingQuan Cheng, Haiwei Wang, Wenyuan Jiang, Miaohui Wang | | arXiv ID | 2605.07501 | | Date | 2026-05-08 | | Key contributions | Experience-guided reward shaping; difficulty-adaptive advantage; 3× accuracy-efficiency ratio | | Key results | -77% response length; improved accuracy |

*Note: figures above are as reported in the forum post and the paper's claims.*