How Fast Should a Model Commit to Supervision? Training Reasoning Models Beyond Cold-Start Stalling (GARL & PAFT)

Forum topic · 小凯 · 2026-04-30

Summary

This paper (arXiv 2504.21150, by Chu-Cheng Lin and Eugene Ie) addresses cold-start stalling when post-training reasoning models with only output-level supervision. Under reinforcement learning from verifiable rewards (RLVR), adaptation stalls when the initial success probability p0 is small. The authors use the Tsallis q-logarithm to define a loss family J_Q that interpolates between RLVR (q=0, the exploitation pole) and log-marginal-likelihood over latent trajectories (q=1, the density-estimation pole). All members share the same per-example gradient direction and differ only by a scalar amplification factor P_theta^{-q} that reweights instances independently of the learning rate. Under gradient flow, the exploitation pole needs Omega(1/p0) time to escape cold start, while the density-estimation pole needs only Theta(log(1/p0)); intermediate q values trade off escape speed against noise memorization. Two Monte Carlo estimators are derived: Gradient-Amplified RL (GARL) and Posterior-Annealed Fine-Tuning (PAFT). On FinQA, HotPotQA, and MuSiQue, GARL with q=0.75 escapes cold start where GRPO fails entirely, and PAFT with q=0.75 yields stable training, reaching 47.9 maj@16 on HotPotQA, 14.4 points above GRPO.

Paper Overview

Field: Machine Learning Authors: Chu-Cheng Lin, Eugene Ie Published: 2026-04-29 arXiv: 2504.21150

Summary in English

When post-training reasoning models with only output-level supervision, reinforcement learning from verifiable rewards (RLVR) stalls when adapting to new tasks with a small initial success probability p0. Using the Tsallis q-logarithm, the authors define a loss family J_Q that interpolates between RLVR (q=0, the exploitation pole) and the log-marginal-likelihood over latent trajectories (q=1, the density-estimation pole).

All members of the family share the same per-example gradient direction, differing only by a scalar amplification factor P_theta^{-q} that reweights each instance independently of the learning rate. This amplification is the mechanism that resolves cold-start stalling: under gradient flow, the exploitation pole requires Omega(1/p0) time to escape cold start, whereas the density-estimation pole needs only Theta(log(1/p0)); intermediate q values trade off escape speed against noise memorization.

Since P_theta is intractable, two Monte Carlo estimators are derived from two factorizations of the gradient:

GARL (Gradient-Amplified RL): samples from the prior and amplifies the RL gradient. Lower variance.
PAFT (Posterior-Annealed Fine-Tuning): importance-resamples from the posterior and runs standard SFT. More coherent gradient semantics.

Both estimators have bias O(q / (M * P_theta^{q+1})).

Experimental Results

On FinQA, HotPotQA, and MuSiQue:

With q = 0.75, GARL significantly mitigates cold-start stalling and successfully escapes cold start on tasks where GRPO fails entirely.
Under warm-start conditions, low-q GARL performs best on the training-stable FinQA.
On HotPotQA and MuSiQue, GARL training is unstable, while PAFT with q = 0.75 provides stable gradients, achieving up to 47.9 maj@16 on HotPotQA, 14.4 points higher than GRPO.

Original Abstract

Adapting reasoning models to new tasks during post-training with only output-level supervision stalls under reinforcement learning from verifiable rewards (RLVR) when the initial success probability p0 is small. Using the Tsallis q-logarithm, we define a loss family J_Q that interpolates between RLVR (at q=0, the exploitation pole) and the log-marginal-likelihood over latent trajectories (at q=1, the density-estimation pole). All members share the same per-example gradient direction, differing only by a scalar amplification P_theta^{-q} that reweights each instance independently of the learning rate.

---

*Auto-collected on 2026-04-30*