PreRL Explained: Moving AI Reasoning From P(y|x) to P(y) With Dual-Space RL

Forum topic · 小凯 · 2026-04-16

Summary

This Chinese forum post offers a detailed, Feynman-inspired walkthrough of PreRL (Pre-train Space Reinforcement Learning), a method that shifts reinforcement learning for LLM reasoning from optimizing conditional probability P(y|x) to the marginal distribution P(y). The author argues that standard RLVR is fundamentally bounded by the base model's output distribution, like a fisherman who only catches fish at GPS coordinates. PreRL instead optimizes the model's ability to generate reasonable reasoning trajectories in general. A key finding: positive-sample reinforcement (PSR) in pre-train space causes on-policy learning collapse and hurts performance, while negative-sample reinforcement (NSR) boosts transition thoughts 14.89x and reflection thoughts 6.54x. The proposed two-stage DSRL (Dual Space RL) pipeline—NSR-PreRL followed by standard RL—outperforms GRPO across AMC, MATH500, AIME24/25, Minerva, and OlympiadBench (57.54 vs 55.79 average on Qwen3-4B), with gradient-alignment experiments (dot product non-negative, mean cosine similarity 0.44) supporting the theory. The post also raises caveats: limits of the analogy, over-penalization risks, unobservability of P(y), and compute costs.

PreRL: Teaching AI to Think — From P(y|x) to P(y)

*This post is an English translation of a Chinese tech-forum article analyzing the PreRL (Pre-train Space Reinforcement Learning) paper. The original uses a fishing metaphor and Richard Feynman's philosophy to explain the method.*

The Fisherman's Epiphany

Imagine a fisherman guided by GPS: the navigator says fish are at coordinate X, so he casts his net and catches Y. Over time he becomes excellent at fishing in known waters. This is what today's AI does: given question X, output answer Y — optimizing P(y|x), the conditional probability of answer Y given input X.

But an old fisherman says: "You only fish where the GPS tells you. A real fisherman knows the sea itself — currents, seasons, migration patterns. Understand the sea, and you can fish anywhere."

That's PreRL's core insight: rather than teaching AI tricks for specific questions, teach it the structure of the world itself.

Why P(y|x) Isn't Enough

Traditional RLVR (reinforcement learning with verifiable rewards), as used by DeepSeek-R1, works like a student who memorizes solutions to every exercise type — great at finding correct answers among things it already knows, but unable to go beyond its training data.

The paper states the blunt truth: the capability of existing RLVR is fundamentally bounded by the base model's output distribution. RL *uncovers* existing abilities; it doesn't create new ones. Like treasure-hunting on an island: no matter how hard you dig, you can't find treasure beyond the shore. PreRL aims to enlarge the island.

Formal Definitions

Definition 1 (Post-training space optimization): optimize the conditional policy πθ(y|x), with learning objectives strictly based on specific input queries x ~ X.
Definition 2 (Pre-train space optimization): optimize the marginal policy πθ(y), directly optimizing the intrinsic distribution of reasoning trajectories.

The paper also proves a gradient alignment theorem: the dot product of ∇θ log πθ(y) and ∇θ log πθ(y|x) is always non-negative, meaning optimizing the marginal distribution also optimizes the conditional one — the theoretical foundation for PreRL.

Positive vs Negative Sample Reinforcement: A Surprising Finding

PSR Can Be Harmful

Intuition says: reward correct answers. But in pre-train space, positive-sample reinforcement (PSR) *degrades* performance by causing on-policy learning collapse — the model repeatedly reinforces what it already gets right instead of exploring. Figure 3(a) shows PSR-PreRL successfully raises conditional probability πθ(y|x) yet final performance drops.

NSR: The 14.89x Miracle

The most striking result: punishing wrong reasoning paths stimulates the model's endogenous reasoning ability:

Transition thoughts increased 14.89x
Reflection thoughts increased 6.54x

Like a strict coach who never praises but always points out mistakes, negative feedback forces the model to develop internal self-correction mechanisms — reflection. The bicycle-learning analogy: PSR gives candy every time you ride straight (you learn only flat-road straight lines); NSR steadies you when you wobble but gives no candy (you learn balance, turning, braking, even tricks).

DSRL: Dual Space RL, a Two-Stage Approach

1. Stage 1 — NSR-PreRL: explore the "knowledge ocean" with negative-sample reinforcement: prune obviously wrong reasoning paths, stimulate reflection and exploration, expand the cognitive boundary. Figure 1(c) shows DSRL beats GRPO in training dynamics with continuously growing response length. 2. Stage 2 — Standard RL: once broad exploration and reflection exist, polish with conventional P(y|x) optimization.

The authors call this Policy Reincarnation: the model is first "reborn" into pre-train space, endures NSR trials, then reincarnates into post-training space with new capabilities.

Experimental Results

| Model | AMC | MATH500 | AIME24 | AIME25 | Minerva | OlympiadBench | Avg | |-------|-----|---------|--------|--------|---------|---------------|-----| | Qwen3-4B Vanilla | 68.28 | 80.17 | 23.13 | 20.00 | 23.62 | 32.33 | 41.26 | | GRPO | 87.81 | 89.17 | 46.46 | 40.94 | 30.06 | 40.29 | 55.79 | | DSRL (PreRL+RL) | 89.22 | 89.68 | 51.15 | 43.44 | 30.48 | 41.26 | 57.54 |

DSRL beats GRPO on every benchmark (+1.75 average). More importantly, Pass@K improves, indicating genuinely stronger exploration ability. Gradient-alignment experiments support the theory: gradient dot product mean +9.2 with 100% of samples non-negative; cosine similarity mean 0.44; log-probability differences concentrated in 0.1–0.4.

Deeper Implications

Teaching vs learning: RLVR "teaches" AI what's correct; PreRL lets AI *learn* by exploring and recovering from errors.
Negative samples may be more valuable than positive ones — errors define boundaries, stimulate exploration, and build resilience.
The pre-training/fine-tuning boundary is blurring: pre-training can be online, exploratory, and trial-driven.
The nature of reasoning: not knowledge, but the ability to explore the error space and recover from it.

Critiques Raised in the Post

Where does the metaphor fail? The ocean's laws are objective; "what counts as reasonable reasoning" depends on human definitions — who draws the boundaries of the pre-train-space sea?
Where does NSR stop? Over-penalization could make the model overly conservative — punishing errors while preserving the courage to err is an art of balance.
Is P(y) really optimized? P(y) is unobservable; we infer it indirectly from downstream performance — a methodological open question.
Compute cost: two-stage training is more expensive than single-stage; industrial cost-benefit analysis is still needed.

Conclusion

From P(y|x) to P(y) is not just a mathematical tweak but a cognitive paradigm shift — from "answering questions" to "understanding the world." NSR-PreRL embodies the Feynman ethos: embrace mistakes, learn from them, never stop being curious. The goal is an AI that is a navigator of the knowledge ocean, not a fisherman waiting for GPS coordinates.

References

1. Tan, Y., Wang, M., Liu, B., et al. (2026). Pre-train Space Reinforcement Learning: From P(y|x) to P(y). *arXiv preprint arXiv:2604.14142*. 2. Feynman, R. P. (1985). *Surely You're Joking, Mr. Feynman!* W. W. Norton & Company. 3. Feynman, R. P. (1998). *The Meaning of It All: Thoughts of a Citizen-Scientist*. Perseus Books. 4. Guo, S., et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. *arXiv preprint*. 5. Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. *arXiv preprint*. 6. Sutton, R. S., & Barto, A. G. (1998). *Reinforcement Learning: An Introduction*. MIT Press. 7. Schulman, J., et al. (2015). Trust Region Policy Optimization. *ICML*.