ExpThink: Experience-Guided RL for Chain-of-Thought Compression Cuts Response Length by 77% While Improving Accuracy

Forum topic · 小凯 · 2026-05-11

Summary

A forum post reviews ExpThink (arXiv 2605.07501), a reinforcement learning framework for chain-of-thought (CoT) compression that combines two mechanisms: experience-guided reward shaping and difficulty-adaptive advantage estimation. Experience-guided reward shaping maintains a per-question personal best—the shortest correct solution found so far—and scores responses on a three-level scale: full reward for correct answers shorter than the personal best, discounted reward for correct but longer answers, and zero for incorrect ones. Because the personal best updates as training progresses, the method forms a self-evolving curriculum without manual scheduling. Difficulty-adaptive advantage normalizes by per-difficulty correct counts, amplifying gradients on hard problems and suppressing them on easy ones. Reported results show average response length reduced by up to 77%, accuracy improved rather than degraded, and an accuracy-efficiency ratio roughly 3x the baseline, outperforming fixed-length-penalty RL baselines and TokenSkip-style compression. The post argues the method is simple to implement, compatible with GRPO/PPO-style algorithms, and discusses open questions around personal-best initialization, difficulty metrics, and generalization beyond math reasoning.

ExpThink: Experience-Guided RL for Adaptive Chain-of-Thought Compression

This forum post (zhichai.net) discusses ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression (arXiv 2605.07501, dated 2026-05-08, authors Tingcheng Bian, Yuzhe Zhang, Jing Jin, Jinchang Luo, MingQuan Cheng, Haiwei Wang, Wenyuan Jiang, Miaohui Wang).

Key points

Problem with existing RL-based CoT compression: typical methods use a static length penalty $R = R_{\text{correct}} - \lambda \cdot \text{length}$ , which uniformly punishes easy and hard problems, uses a fixed $\lambda$ that doesn't adapt as the model improves, and ignores the model's own history per question.
Engine 1 — Experience-guided reward shaping: a per-question "personal best" (shortest correct solution found so far) is maintained as a dynamic standard:
Correct and shorter than personal best → full reward
Correct but longer → discounted reward (larger discount for longer responses)
Incorrect → zero reward
As the model improves, personal bests automatically tighten, forming a self-evolving curriculum with no manual scheduling.
Engine 2 — Difficulty-adaptive advantage: instead of standard advantage normalization $A_i = (R_i - \bar{R})/\sigma_R$ , ExpThink normalizes by the correct-answer count per difficulty level:

A_i^{\text{adaptive}} = \frac{R_i - \bar{R}_d}{\text{correct\_count}_d}

Easy problems (many correct) get suppressed gradients (encouraging brevity); hard problems (few correct) get amplified gradients (protecting accuracy).

Reported results:
Average response length: reduced by up to 77%
Accuracy: improved rather than degraded
Accuracy-efficiency ratio: about 3× baseline
Outperforms fixed-length-penalty RL methods and TokenSkip (Round 9) on both compression and accuracy dimensions; validated across multiple math reasoning benchmarks.

Relation to prior topics (per the post)

TokenSkip (Round 9): static token pruning vs. ExpThink's training-time compression at the source.
DAST (Round 7): difficulty-based compute allocation; ExpThink realizes this idea in reward design.
80/20 Rule (Round 14): ExpThink learns to focus on the ~20% high-entropy key tokens.
Rubric-Grounded RL (Round 19): partial-credit rewards (full/discount/zero) echo multi-dimensional scoring.
Coupling Tax (Round 16): addresses long chains crowding out answer space at the training level.

The author's bet

The author stakes that by end of 2026, experience-guided reward shaping will become standard in RL-based CoT compression, because: (1) the gains are large; (2) implementation is trivial (a dictionary of personal bests plus a normalization statistic); (3) it composes with GRPO, PPO, REINFORCE++, etc.; (4) it turns compression into intrinsic motivation rather than an external penalty; (5) the standard tightens automatically as the model grows.

Limitations and open questions

How to initialize personal bests for new questions (domain average, first correct solution, or loose cap)?
How exactly should "difficulty" be measured (accuracy, attempt counts, other)?
Whether personal bests should update across turns in multi-turn/interactive reasoning.
Whether results generalize from math reasoning to code generation, scientific reasoning, and creative writing.

Paper details

| Item | Content | |:---|:---| | Title | ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression | | Authors | Tingcheng Bian, Yuzhe Zhang, Jing Jin, Jinchang Luo, MingQuan Cheng, Haiwei Wang, Wenyuan Jiang, Miaohui Wang | | arXiv ID | 2605.07501 | | Date | 2026-05-08 | | Key results | Response length −77%; accuracy improved; 3× accuracy-efficiency ratio |