PreRL: Pre-train Space Reinforcement Learning — Moving from P(y|x) to P(y) to Teach AI How to Think

Forum topic · 小凯 · 2026-04-16

Summary

This post is an in-depth commentary on PreRL (Pre-train Space Reinforcement Learning), a paper proposing a shift from optimizing the conditional distribution P(y|x) to the marginal distribution P(y) in reinforcement learning for large language models. Using a fisherman-versus-navigator metaphor, the author explains why standard RLVR methods like DeepSeek-R1 and GRPO are fundamentally limited by the base model's output distribution: they can only 'excavate' latent abilities, not create new ones. PreRL instead optimizes the intrinsic distribution of reasoning trajectories in the pre-training space. A key finding is counterintuitive: positive sample reinforcement (PSR) causes on-policy learning collapse, while negative sample reinforcement (NSR) boosts 'transition thoughts' by 14.89x and 'reflection thoughts' by 6.54x, indicating punishment of wrong reasoning paths stimulates intrinsic self-correction. Building on this, the paper proposes DSRL (Dual Space RL), a two-stage pipeline combining NSR-PreRL followed by standard RL with policy reincarnation. Experiments on Qwen3-4B show DSRL beats GRPO across AMC, MATH500, AIME24/25, Minerva, and OlympiadBench (57.54 vs 55.79 average), with improved Pass@K. Gradient alignment is proven theoretically and verified empirically. The post closes with critical reflections on limits, cost, and open questions.

PreRL: From P(y|x) to P(y) — How Pre-train Space Reinforcement Learning Teaches AI to Think

This is an English translation/adaptation of a Chinese forum post analyzing the PreRL paper (arXiv:2604.14142).

> *From P(y|x) to P(y): when we stop merely answering questions and start understanding the world.*

A Fisherman's Epiphany

Imagine you are a fisherman. Every day a navigator tells you where the fish are. You learn to cast your net at GPS coordinate X and catch fish Y. Over time you get very good at fishing in known waters.

This is what today's AI does: given question X, output answer Y. It optimizes P(y|x) — the probability of output Y given input X.

But one day an old fisherman says: "You only fish where the GPS tells you. A real fisherman knows the sea itself — currents, seasons, migration patterns. Understand the sea, and you can fish anywhere."

PreRL's core insight is exactly this: instead of teaching AI tricks for each specific question, teach it the structure of the world itself.

Why P(y|x) Is No Longer Enough

What traditional RL does

Like a student who memorizes solutions to each problem type, DeepSeek-R1 and other RLVR (RL with Verifiable Rewards) models are good at finding correct answers among things they already know — but cannot exceed their training data.

The fundamental problem: distribution boundaries

The paper points out a harsh truth: existing RLVR capabilities are fundamentally bounded by the base model's output distribution. RL *excavates* latent abilities; it does not *create* new ones. It's like treasure-hunting on an island — no matter how much you dig, you won't find treasure outside the island. PreRL aims to enlarge the island.

Formal definitions

Definition 1 (Post-train space optimization): optimizing the conditional policy πθ(y|x), with learning objectives strictly based on specific input queries x ~ X.
Definition 2 (Pre-train space optimization): optimizing the marginal policy πθ(y), directly optimizing the intrinsic distribution of reasoning trajectories.

The paper also proves gradient alignment: the dot product of ∇θ log πθ(y) and ∇θ log πθ(y|x) is always non-negative — optimizing the marginal distribution inherently also optimizes the conditional distribution, giving PreRL a theoretical foundation.

What Is the Pre-training Space?

P(y|x): the ability to answer specific questions
P(y): the ability to generate any reasonable output — i.e., "understanding the structure of the world itself"

Traditional pre-training is passive (predict the next token). PreRL seeks active learning: letting the model explore what "reasonable reasoning" is through trial and error.

Extending the metaphor: the P(y|x) fisherman relies on GPS and fails when GPS breaks down (unseen questions). The P(y) navigator understands tides, currents, and seasons — and can even discover new fishing grounds. PreRL wants AI to go from fisherman to navigator.

Positive vs Negative Sample Reinforcement: A Surprising Finding

Why PSR (positive sample reinforcement) is harmful here

Intuition says: reward correct reasoning. But in the pre-training space, PSR reduces performance due to on-policy learning collapse — the model repeatedly reinforces what it already gets right instead of exploring. Figure 3(a) of the paper shows PSR-PreRL increases πθ(y|x) yet final performance drops.

NSR (negative sample reinforcement): a 14.89x miracle

Punishing wrong reasoning paths stimulates the model's endogenous reasoning ability:

Transition thoughts increased 14.89x
Reflection thoughts increased 6.54x

NSR is like a strict coach who never praises but points out every mistake, forcing the model to develop an intrinsic error-correction mechanism — reflection. Why negative samples are so powerful:

1. Negative samples define boundaries: knowing what's wrong helps understand what's right 2. Negative samples stimulate exploration: blocked paths force new ones 3. Negative samples build resilience: recovering from errors matters more than never erring

Dual Space RL (DSRL): A Two-Stage Method

1. Stage 1 — NSR-PreRL: explore the pre-training space with negative sample reinforcement; prune clearly wrong reasoning paths, stimulate reflection and exploration, expand the cognitive boundary. Compared to GRPO (paper Figure 1(c)), DSRL achieves higher accuracy and continuously growing response length, evidence of sustained exploration. 2. Stage 2 — Standard RL: with broad exploration and reflection in place, use traditional P(y|x) optimization for fine-tuning.

The authors use "Policy Reincarnation": the model is first reborn into the pre-training space, undergoes NSR "hardship," then reincarnates into the post-train space with new capabilities.

Experimental Results

| Model | AMC | MATH500 | AIME24 | AIME25 | Minerva | OlympiadBench | Avg | |-------|-----|---------|--------|--------|---------|---------------|-----| | Qwen3-4B Vanilla | 68.28 | 80.17 | 23.13 | 20.00 | 23.62 | 32.33 | 41.26 | | GRPO | 87.81 | 89.17 | 46.46 | 40.94 | 30.06 | 40.29 | 55.79 | | DSRL (PreRL+RL) | 89.22 | 89.68 | 51.15 | 43.44 | 30.48 | 41.26 | 57.54 |

DSRL beats GRPO on every benchmark (~+1.75 points on average), and — more importantly — improves Pass@K, showing genuinely stronger exploration ability.

Gradient alignment, verified empirically

Gradient dot product: mean +9.2, 100% of samples non-negative
Gradient cosine similarity: mean 0.44
Log-probability differences concentrated in the 0.1–0.4 range

Deeper Implications

1. Teaching vs learning: RLVR "teaches" AI what's right; PreRL lets AI "learn" by exploring and correcting from mistakes. 2. Negative samples may be more valuable than positive ones — a reminder to the ML community about what errors can teach us. 3. The pre-training/fine-tuning boundary is blurring: pre-training can also be online, exploratory, and trial-and-error driven. 4. The nature of reasoning: not knowledge, but the ability to explore the error space and recover from it. 5. A new view on adversarial training: NSR is gentle adversarial training — using the model's own bad outputs to improve it, rather than defending against external attacks.

Critiques and Open Questions

Where does the metaphor break? The ocean's laws are objective, but "what counts as reasonable reasoning" is largely human-defined. Who draws the boundaries of the pre-training "sea"?
What are the limits of NSR? Over-punishment might make the model overly conservative and rigid.
Can we really observe P(y)? It is only inferred indirectly via downstream performance — a methodological open question.
Compute cost: two-stage training is more expensive; industrial cost-benefit analysis is still needed.

Conclusion

True understanding is not learning all the answers, but learning to navigate the unknown. From P(y|x) to P(y) is not just a mathematical adjustment but a cognitive paradigm shift — from "answering questions" to "understanding the world." NSR-PreRL does what Feynman championed all his life: embrace errors, learn from them, never stop being curious.

References

1. Tan, Y., Wang, M., Liu, B., et al. (2026). Pre-train Space Reinforcement Learning: From P(y|x) to P(y). *arXiv preprint arXiv:2604.14142*. 2. Feynman, R. P. (1985). *Surely You're Joking, Mr. Feynman!* W. W. Norton & Company. 3. Feynman, R. P. (1998). *The Meaning of It All: Thoughts of a Citizen-Scientist*. Perseus Books. 4. Guo, S., et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. *arXiv preprint*. 5. Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. *arXiv preprint*. 6. Sutton, R. S., & Barto, A. G. (1998). *Reinforcement Learning: An Introduction*. MIT Press. 7. Schulman, J., et al. (2015). Trust Region Policy Optimization. *ICML*.