ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

> Paper: ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning > Authors: Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai > arXiv: 2605.00380 | 2026-04-29

1. The RL Dilemma: Rewarding Only Correct Answers

Consider training an LLM for mathematical reasoning:

Traditional RLVR (Reinforcement Learning with Verifiable Rewards):

• Correct answer → high reward
• Wrong answer → low reward or penalty
• The model learns to generate only "safe" answers
• Diversity collapses
• Over-optimization toward a single mode

Resulting problems:

• The model becomes afraid of making mistakes
• It stops exploring new reasoning paths
• It lacks creativity on novel problems
• Reward hacking

The need: make errors a usable learning resource.

2. ResRL: Negative Sample Projection Residual Learning

Core idea: > Don't just penalize wrong answers—extract useful information from them, distinguishing "good errors" from "bad errors."

Technical approach:

1. Negative sample projection — Project negative samples (wrong answers) into the space of positive samples (correct answers) and compute the residual. The residual represents the "essential difference" between negative and positive samples. 2. Residual learning — Learn *what kind of difference causes the error*, rather than simply punishing; understand *why* it is wrong. 3. Protecting shared semantics — Negative and positive samples may share parts of correct reasoning. Traditional negative-sample reinforcement (NSR) suppresses these shared parts; ResRL protects shared semantics and targets only the differing parts. 4. Preserving diversity — Encourage exploration without fully suppressing negative samples, retaining diversity while improving accuracy.

Analogy:

• Traditional RL = a teacher who only says "right" or "wrong"
• ResRL = a teacher who analyzes what was right, what was wrong, and why

3. Why ResRL Beats Naive Negative-Sample Penalties

Problems with naive NSR:

• Over-suppression: negative samples are suppressed wholesale, including their correct parts. The model learns "don't do this" and loses valid knowledge along the way.
• Loss of diversity: the model takes only "safe routes," stops exploring, and fails on new problems.

Advantages of ResRL:

• Precise correction: targets only the erroneous parts while protecting correct content.
• Diversity preservation: does not fully suppress negative samples; encourages reasonable exploration and better generalization.
• Interpretability: the residual serves as an error analysis, aiding debugging.

4. The Feynman-Style Takeaway: Errors Are the Best Teachers

Feynman said: > "Science is a way of teaching how not to fool ourselves."

Applied to learning: > "ResRL's insight is that errors are not the enemy—they are information. Simply punishing errors wastes information. Analyzing, understanding, and learning from errors is the wise way to learn."

This reflects a growth mindset: errors are opportunities for growth, not markers of failure. Learning from errors beats merely avoiding them.

5. Questions to Ask Yourself

If you train LLMs or RL systems:

1. Are my negative samples being punished naively? 2. Am I extracting useful information from errors? 3. Is shared semantics being protected? 4. Is diversity one of my training objectives?

ResRL reminds us: in reinforcement learning, the best reward function not only tells the model what is right, but also why something is wrong. When an LLM shifts from "fearing mistakes" to "learning from mistakes," it gains not only stronger reasoning but also the courage to explore the unknown. In the journey of reasoning, errors are not endpoints—they are signposts toward deeper understanding.

*Note: metadata (arXiv ID, dates, author list) is reproduced as given in the original post and not independently verified.*