MRT: Redefining LLM Test-Time Compute Efficiency via Cumulative Regret and Meta-RL

Forum topic · 小凯 · 2026-05-11

Summary

Researchers from Carnegie Mellon University and Hugging Face (arXiv:2503.07572, March 2025) reformulate LLM test-time compute optimization as a meta-reinforcement learning problem and propose Meta Reinforcement Fine-Tuning (MRT). By splitting an LLM's output stream into episodes and defining cumulative regret as the gap to the optimal policy at each budget, their analysis shows current long chain-of-thought reasoning models often fail to improve accuracy with additional episodes—majority voting over truncated chains can even outperform full long CoT under matched FLOPs. MRT trains models with an episode-level progress reward, a tractable surrogate for regret minimization that requires no LLM judge. On 1.5B-parameter models (DeepScaleR, DeepSeek-R1-Distill-Qwen), MRT achieves roughly 2-3x relative improvement over GRPO on AIME 2024/2025 and AMC 2023, plus 1.5x better token efficiency, and extrapolates stably to doubled test budgets where outcome-reward RL baselines accumulate growing regret. The work argues test-time compute should be optimized for per-episode progress rather than raw chain length, addressing over-commitment to fixed token budgets.

Key points

A March 2025 study by Yuxiao Qu, Amrith Setlur, Lewis Tunstall, Ruslan Salakhutdinov, Aviral Kumar and colleagues from Carnegie Mellon University and Hugging Face (arXiv:2503.07572) reformulates LLM test-time compute optimization as a meta-reinforcement learning problem and proposes Meta Reinforcement Fine-Tuning (MRT).

The problem: test-time compute is used inefficiently

Long-CoT "slow thinking" models (DeepSeek-R1, OpenAI o1) exhibit two flaws: redundant generation on easy problems and poor exploration on hard ones.
Analysis of DeepSeek-R1-Distill-Qwen-32B on OmniMATH and AIME 2024 shows that when chains of thought contain 41–45 episodes, additional episodes do not raise—and sometimes lower—accuracy. Under FLOPs-matched evaluation, majority voting over truncated CoTs often beats the full long CoT.
Standard outcome-reward RL with a fixed token budget $$C_0$$ causes over-commitment: models fail when deployment budgets are smaller and cannot exploit extra compute when larger.

The formalization: episodes and cumulative regret

The output stream is split into episodes $\mathbf{z} = [\mathbf{z}_0, \dots, \mathbf{z}_{k-1}]$ ; the LLM acts as the learning algorithm and each episode is an adaptation step.
Cumulative regret measures the gap between the policy's conditional success probability (via a meta-prover $\mu$ ) and the optimal $$j$$ -episode policy $\pi_j^*$ :

\Delta_k^{\mu}(\mathbf{x}; \pi) := \mathbb{E}_{\mathbf{z} \sim \pi(\cdot|\mathbf{x})} \left[ \sum_{j=0}^{k-1} J_r(\mathbf{x}; \pi_j^*) - J_r(\mathbf{x}; \mu(\cdot|\mathbf{x}, \mathbf{z}_{0:j})) \right]

Intuition: every new episode should increase the probability of answering correctly.

The method: progress rewards

Since $\pi_j^*$ is unknown, MRT optimizes an episode-level progress reward—the marginal contribution of an episode to the meta-prover's success probability—requiring no LLM judge.
The training objective combines the outcome reward with a weighted sum of progress bonuses, using $\pi_{\text{old}}$ both as a trust-region constraint and to avoid branched rollouts, so MRT runs on standard RL infrastructure.
Unlike SCoRe or step-level progress work (Setlur et al., 2024), MRT computes progress across episodes, naturally fitting long-CoT models where each "think" block is one episode.

Results

1.5B models (DeepScaleR-1.5B-Preview, DeepSeek-R1-Distill-Qwen-1.5B): MRT reaches ~35–40% on AIME 2024, ~30–35% on AIME 2025, ~80–85% on AMC 2023 — a 2–3x relative gain over GRPO, and 1.5x better token efficiency (5x vs. the base model).
On a backtracking task with Llama3.1, MRT is 1.6–1.7x more token-efficient than STaR and GRPO.
Budget extrapolation: MRT-trained models keep stable progress and low regret when extrapolating to 2x the training budget, whereas outcome-reward RL models accumulate growing regret—i.e., they cannot use extra test-time compute effectively.

Deeper insight

Outcome-reward RL teaches models to *sometimes get the answer right*, not to *approach the answer progressively across episodes*. On hard problems this leads to repeated exploitation of known strategies rather than effective exploration—consistent with classic meta-RL findings on generalization (Duan et al., 2016; Wang et al., 2016), here transferred from robotics to LLM reasoning.

Limitations and open questions

The meta-prover $\mu$ is the same LLM forced to answer by terminating "think" blocks; robustness when $\mu \neq \pi$ is unverified.
Episode segmentation currently uses a fixed count plus natural-language cues ("Wait", "Alternatively"); automatic segmentation for tree search or parallel verification remains open.
Combining SFT warm-starts with MRT's dense rewards for faster convergence is unexplored.

Conclusion

MRT reframes test-time compute optimization through regret minimization: more computation is not inherently better—every step must make progress. This budget-agnostic view has direct implications for deploying reasoning models under real compute constraints.

Paper details

| Item | Detail | |:---|:---| | Title | Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning | | Authors | Yuxiao Qu, Matthew Y.R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, Aviral Kumar | | Institutions | Carnegie Mellon University, Hugging Face | | arXiv | 2503.07572 (2025-03-10) | | Models | DeepScaleR, DeepSeek-R1-Distill-Qwen (1.5B/7B), Llama3.1 | | Benchmarks | AIME 2024/2025, AMC 2023, OmniMATH, MATH |