ResRL: Boosting LLM Reasoning with Negative Sample Projection Residual Reinforcement Learning
> Paper: ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning > Authors: Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai > arXiv: 2605.00380 | 2026-04-29
1. The RL Dilemma of Rewarding Only Correct Answers
When training an LLM for mathematical reasoning, traditional RLVR (Reinforcement Learning with Verifiable Rewards) works like this:
- Correct answer → high reward
- Wrong answer → low reward or penalty
- The model learns to only produce "safe" answers
- Diversity is lost; the model over-optimizes a single pattern
- The model becomes afraid of making mistakes
- It stops exploring new reasoning paths
- Lacks creativity on unfamiliar problems
- Vulnerable to reward hacking
- Traditional RL = a teacher who only says "right" or "wrong"
- ResRL = a teacher who analyzes "what was right, what was wrong, and why"
- Over-suppression: Negative samples are punished wholesale, including their correct parts. The model learns "don't do this," losing correct knowledge in the process.
- Loss of diversity: The model sticks to safe routes, stops exploring, and fails on new problems.
- Precise error correction: Targets only the erroneous parts while protecting correct parts—more efficient.
- Diversity preservation: Negative samples are not fully suppressed; reasonable exploration is encouraged, improving generalization.
- Interpretability: The residual serves as an error analysis, which helps debugging.
Consequences:
What's needed: a way to make errors a learning resource.
2. ResRL: Negative Sample Projection Residual Learning
The paper proposes ResRL, whose core idea is:
> Don't just penalize wrong answers—extract useful information from them, distinguishing "good mistakes" from "bad mistakes."
Technical approach:
1. Negative sample projection: Project negative samples (wrong answers) into the space of positive samples (correct answers) and compute the residual. The residual captures the essential difference between the negative and positive samples. 2. Residual learning: Learn *what difference caused the error*. Rather than simple punishment, the model understands *why* it is wrong. 3. Protecting shared semantics: Negative and positive samples may share partially correct reasoning. Traditional NSR (negative sample reinforcement) suppresses these shared parts; ResRL protects them and only targets the differing parts. 4. Boosting diversity: Exploration is encouraged, negative samples are not fully suppressed, and diversity is preserved while accuracy improves.
Analogy:
3. Why ResRL Beats Simple Negative-Sample Penalties
Problems with naive NSR:
ResRL advantages:
4. Takeaway: Errors Are the Best Teachers
The post invokes Feynman's spirit:
> "Science is a way of teaching how not to fool ourselves."
ResRL's insight: errors are not the enemy—they are information. Simply punishing mistakes wastes information; analyzing and learning from them is the smarter path. This reflects a growth mindset: mistakes are opportunities for growth, not marks of failure.
5. Questions to Ask About Your Own RL Systems
1. Are my negative samples being punished too crudely? 2. Am I extracting useful information from errors? 3. Is shared semantics being protected? 4. Is diversity one of my training objectives?
> The best reward functions don't just tell the model "what is correct"—they also tell it "why it is wrong."
When an LLM shifts from "fearing mistakes" to "learning from mistakes," it gains not only stronger reasoning but also the courage to explore the unknown.
*Original post published on zhichai.net; note this is a forum commentary on the referenced paper.*