The Sparsity Puzzle of RLVR: The Three-Gate Theory and the Ridge-vs-Valley Metaphor

Forum topic · ✨步子哥 · 2025-12-15

Summary

This forum post explores why reinforcement learning with verifiable rewards (RLVR) produces extremely sparse parameter updates when improving reasoning and coding abilities, unlike the dense updates seen in supervised fine-tuning (SFT). It introduces a 'three-gate theory' explaining this sparsity: (1) the KL anchor, a single-step policy-KL constraint that keeps updates close to the base policy; (2) model geometry, which steers updates toward low-curvature, spectrum-preserving directions that avoid the model's principal directions; and (3) numerical precision, where bfloat16 acts as a lens that hides micro-updates and amplifies the apparent sparsity. The post illustrates the difference with a geometric metaphor: SFT climbs high-curvature 'ridge' directions, causing dramatic spectral drift and altering core knowledge, while RLVR walks along gentle 'valley' paths off the principal subspace, preserving core structure. Practical implications follow: LoRA, which naturally updates non-principal directions, fits RLVR well, whereas PiSSA—which targets principal singular directions—forces the model up the ridge and tends to cause training collapse in RL tasks. Experiments cited in the post show PiSSA underperforms plain LoRA under RLVR, suggesting SFT and RL require distinct parameter-efficient fine-tuning strategies.

The Sparsity Puzzle of RLVR

When reinforcement learning improves a model's reasoning and coding abilities, the parameter updates turn out to be extremely sparse — like a pianist playing a masterpiece by moving only her pinky finger. RLVR is a paradox: a costly, high-reward training process that changes only a tiny fraction of parameters. This sparsity is not random; it is determined by the model's intrinsic geometry. In contrast, SFT updates are dense.

The Three-Gate Theory

The sparsity of RLVR can be explained by three consecutive constraints ("gates") on parameter updates:

1. Gate 1: KL Anchor — RL induces a single-step policy-KL constraint that keeps updates close to the base policy, limiting the magnitude of parameter changes. 2. Gate 2: Model Geometry — Updates are steered toward low-curvature, spectrum-preserving directions. This is a data-invariant property that forces the model to avoid the "principal directions." 3. Gate 3: Precision — bfloat16 format acts as a lens that hides micro-updates, making the underlying pattern manifest as clear sparsity.

Ridge vs. Valley: A Geometric Metaphor

SFT and RLVR take completely different paths through parameter space:

Ridge (SFT path): Climbing steep, high-curvature "principal directions," causing dramatic spectral drift and altering the model's core knowledge structure.
Valley (RLVR path): Hiking through gentle "off-principal" valleys, preserving the core knowledge structure and enabling efficient, safe learning.

Practical Lessons: LoRA vs. PiSSA

LoRA is naturally suited to RL. It naturally updates non-principal directions, matching RLVR's "valley path." Learning in a low-rank space without disturbing the core geometry lets it stably improve reasoning.
PiSSA is the "mountaineer" in RL. PiSSA focuses on updating the principal singular directions — forcing the model up the ridge. In RL tasks this violates RLVR's basic optimization principles and leads to training collapse.

Experiments show PiSSA not only fails to beat plain LoRA under RLVR but is more prone to training collapse because it forces the model onto the "mountain" path. The takeaway: RL and SFT require different parameter-efficient fine-tuning strategies.