How Fast Should a Model Commit to Supervision? Tsallis q-Loss Family to Fix Cold-Start Stalling in RLVR

Forum topic · 小凯 · 2026-04-30

Summary

A new arXiv paper (2504.21150) by Chu-Cheng Lin and Eugene Ie addresses the cold-start stalling problem in reinforcement learning from verifiable rewards (RLVR), where reasoning models fail to adapt to new tasks under output-level supervision when the initial success probability p0 is small. The authors define a loss family J_Q using the Tsallis q-logarithm, interpolating between RLVR (q=0, exploitation pole) and log-marginal-likelihood over latent trajectories (q=1, density-estimation pole). All loss members share the same per-example gradient direction, differing only by a scalar amplification factor P_theta^{-q} that reweights instances independently of the learning rate. Under gradient flow, the exploitation pole needs Omega(1/p0) time to escape cold start, versus Theta(log(1/p0)) for the density-estimation pole. Two Monte Carlo estimators are derived: GARL (Gradient-Amplified RL) and PAFT (Posterior-Attenuated Fine-Tuning). Experiments on FinQA, HotPotQA, and MuSiQue show q=0.75 variants significantly mitigate cold-start stalling, with PAFT achieving 47.9 maj@16 on HotPotQA, 14.4 points above GRPO.

Paper Overview

Research area: Machine Learning Authors: Chu-Cheng Lin, Eugene Ie arXiv: 2504.21150

Problem

When post-training reasoning models with only output-level supervision, reinforcement learning from verifiable rewards (RLVR) stalls when adapting to new tasks whose initial success probability p0 is small — the classic cold-start stalling problem.

Method

Using the Tsallis q-logarithm, the authors define a loss family J_Q that interpolates between two poles:

q = 0: RLVR (the exploitation pole)
q = 1: log-marginal-likelihood over latent trajectories (the density-estimation pole)

All members of the family share the same per-example gradient direction, differing only by a scalar amplification factor P_theta^{-q} that reweights each instance independently of the learning rate.

Theoretical result on escape time

Under gradient flow:

The exploitation pole (q=0) requires Omega(1/p0) time to escape the cold start.
The density-estimation pole (q=1) requires only Theta(log(1/p0)).
Intermediate q values trade off escape speed against noise memorization.

Two Monte Carlo estimators

Since P_theta is intractable, two estimators are derived from two factorizations of the gradient, both with bias O(q / (M * P_theta^{q+1})):

GARL (Gradient-Amplified RL): samples from the prior and amplifies the RL gradient; lower variance.
PAFT (Posterior-Attenuated Fine-Tuning): importance-resamples from the posterior and runs standard SFT; more coherent gradient semantics.

Results

Experiments on FinQA, HotPotQA, and MuSiQue:

Under cold start, q=0.75 GARL significantly mitigates stalling and escapes the cold start where GRPO fails entirely.
Under warm start, low-q GARL performs best on the training-stable FinQA.
On HotPotQA and MuSiQue, GARL training is unstable, while q=0.75 PAFT provides stable gradients — reaching up to 47.9 maj@16 on HotPotQA, 14.4 points above GRPO.