ExpThink: Experience-Guided RL Framework for Adaptive Chain-of-Thought Compression

In May 2026, Bian et al. proposed ExpThink, a reinforcement learning framework for adaptive Chain-of-Thought (CoT) compression built on two mechanisms: experience-guided reward shaping and difficulty-adaptive advantage estimation. It targets the limitations of uniform, static length penalties in prior RL-based compression methods.

Key points

• Problem: Standard RL compression uses a uniform static penalty R = R_correct − λ·length, which over-compresses easy questions, under-compresses hard ones, never updates as the model improves, and ignores historical best solutions.
• Mechanism 1 — Experience-guided reward shaping: For each question, the framework tracks a personal best B_q (the shortest correct response seen so far) and applies a three-tier reward:
• Correct and record-breaking brevity (length < B_q): full reward R_max
• Correct but longer (length ≥ B_q): discounted reward R_max · B_q / length
• Incorrect: zero
• This creates a self-evolving curriculum: as training progresses, B_q shrinks and the implicit length standard tightens automatically, without manual scheduling.
• Mechanism 2 — Difficulty-adaptive advantage: Standard advantage normalization A_i = (R_i − R̄)/σ_R shares statistics across all questions. ExpThink instead normalizes by cumulative correct count C_d per difficulty level: easy questions get high C_d (suppressed gradients, focusing on compression), hard questions get low C_d (amplified gradients, preserving accuracy). Since C_d correlates monotonically with difficulty, gradient scaling is principled.
• Results:
• Average response length reduced by up to 77%
• Accuracy improved over baseline (unlike fixed-penalty methods, which degrade it)
• 3× accuracy-efficiency ratio, outperforming baselines, fixed penalties, and TokenSkip
• Consistent gains across multiple mathematical reasoning benchmarks

Theoretical analysis

• The experience-guided reward creates a natural compression incentive: the marginal reward w.r.t. length is −R_max·B_q/length², so longer responses receive diminishing marginal rewards.
• Difficulty-adaptive separation lets accuracy optimization dominate on hard questions while compression optimization dominates on easy ones.

Relation to related work

• TokenSkip: post-hoc CoT compression; ExpThink internalizes compression during training, generating shorter reasoning chains from the source.
• DAST: difficulty-adaptive compute at inference; ExpThink adjusts gradients by difficulty at training time — orthogonal dimensions.
• 80/20 Rule: identified critical tokens; ExpThink trains the model to focus on them automatically.
• Rubric-Grounded RL: multi-dimensional scoring; ExpThink's three-tier reward is a partial-credit idea applied to compression.

Limitations and future directions

• Personal-best initialization for new questions (domain statistics, or progressive tightening from the first correct solution).
• Explicit difficulty modeling rather than implicit correct counts.
• Cross-domain transfer to code generation, scientific reasoning, and creative tasks.
• Combination with inference-time optimization (e.g., dynamic routing, evaluation-time deduplication).

Paper details

| Item | Detail | |:---|:---| | Title | ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression | | Authors | Tingcheng Bian, Yuzhe Zhang, Jing Jin, Jinchang Luo, MingQuan Cheng, Haiwei Wang, Wenyuan Jiang, Miaohui Wang | | arXiv ID | 2605.07501 | | Date | 2026-05-08 |

Conclusion: ExpThink shifts CoT compression from "externally penalized compression" to "internally motivated compression," offering a plug-and-play framework compatible with any existing RL algorithm, with simultaneous gains in brevity and accuracy.