The Sparsity Puzzle of RLVR: Three Gates Theory and the Ridge vs. Valley Metaphor
This poster (originally published on zhichai.net) examines a paradox in reinforcement learning with verifiable rewards (RLVR): a high-cost, high-reward training process that only changes a tiny fraction of model parameters. The post argues this sparsity is not random but determined by the model's intrinsic geometry — like a pianist playing a masterpiece by moving only the pinky finger.
Basic Concept: RLVR Sparsity
While reinforcement learning improves reasoning and coding abilities, parameter updates exhibit extreme sparsity. Compared to dense SFT updates, RLVR touches only a small subset of parameters, in a structured, geometry-driven way.
The Three Gates Theory
Each 'gate' constrains parameter updates during RLVR:
1. Gate 1: KL Anchor — RL induces a single-step policy-KL constraint that keeps updates close to the base policy, limiting their magnitude. 2. Gate 2: Model Geometry — Updates are steered toward low-curvature directions that preserve the spectral structure. This data-invariant property forces the model away from 'principal directions.' 3. Gate 3: Precision — bfloat16 acts as a lens, hiding tiny updates and making the underlying pattern appear as clear sparsity.
Ridge vs. Valley: A Geometric Metaphor
- Ridge (SFT path): Climbing steep, high-curvature 'principal directions,' causing dramatic spectral drift and altering the model's core knowledge structure.
- Valley (RLVR path): Hiking through flat, off-principal valleys, preserving core knowledge while enabling efficient and safe learning.
- LoRA: naturally suited to RL. It updates non-principal directions, matching the RLVR 'valley path.' Learning in a low-rank subspace does not damage the core geometry, so reasoning improves stably.
- PiSSA: the 'mountain climber' in RL. It targets principal singular directions — forcing the model up the ridge. In RLVR tasks this violates the optimization principle and leads to training collapse.