Process vs Outcome Reward: The Hard Truth of Reward Design for Agentic RAG Reinforcement Learning
> Paper: Wenlin Zhang et al., "Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning", arXiv:2505.14069v1, 2025-05-20
The Core Question
Imagine teaching a child to do research. You could wait for the final report and grade its quality (Outcome Reward). Or you could give feedback at every step—looking up sources, taking notes, making comparisons (Process Reward). Which works better?
This paper tackles exactly that question—except the "child" is an LLM and the "research" is Agentic RAG.
What the Paper Actually Says
The authors systematically compare two RL training paradigms in agentic RAG:
- Outcome RL: Only the final result matters. The model retrieves documents, generates an answer, and a verifier judges correctness. Correct gets a positive reward; wrong gets a negative one.
- Process RL: Every step gets a reward. Generated a query? Reward. Retrieved relevant documents? Reward. Noticed irrelevant results and switched queries? Bigger reward.
- Outcome RL does fine on simple questions but falls significantly behind Process RL on multi-hop questions
- Process RL needs a carefully designed reward function, or it overfits to surface features
- A hybrid strategy (70% Process + 30% Outcome) performs best on aggregate metrics
A Feynman-Style Note: Naming Is Not Understanding
"Outcome Reward" sounds like "only results matter"; "Process Reward" sounds like "care about the process." Many assume process must be better—if the process is right, results follow.
But that's cargo cult thinking.
The name creates an illusion of "caring about the process." The real problem with Process Reward is: who defines what a "good process" is?
Outcome Reward suffers from sparsity and delay, but at least answer correctness is relatively objective. Process Reward requires pre-defining what's "good" at every step—and that's the danger. You may be encoding your own bias about what a "good research process" looks like, and the model learns to please your bias.
Key Findings
Three hard weaknesses of Outcome RL: 1. Low exploration efficiency: the model needs many tries to stumble on correct answers; early training is painful 2. Gradient conflicts: intermediate-step gradients often fight the final objective's gradient direction 3. Sparse rewards: possibly 50 steps with no signal—severe credit assignment problems
Two dilemmas of Process RL: 1. Process reward design is closer to art than science: how do you score "generated a query"? A query is good only if it eventually leads to the right answer—but you don't know the answer yet. 2. Step-level annotation is extremely expensive: orders of magnitude more labeling work than outcome-level.
Bottom line: a hybrid reward is optimal—process rewards guide early exploration, outcome rewards guarantee final quality.
The Real Insight
The paper's most valuable point isn't "hybrid is best"—that's engineering common sense.
The real insight: reward design for agentic RAG is fundamentally a delayed-gratification problem. The model must balance immediate feedback (process rewards) against long-term goals (outcome rewards)—the same dilemma humans face when learning any complex skill.
Deeper question: when an LLM's research process is shaped by a reward function, is it learning to do research, or learning to please the reward function?
This is Reward Hacking. A poorly designed process reward lets the model find shortcuts—producing intermediate steps that look "correct" but contribute nothing to actually answering the question.
Experimental Data
Experiments on MuSiQue (multi-hop QA) and HotpotQA:
A Critical Perspective
One suspicious assumption: the paper implies an "optimal" reward ratio (70/30). But that ratio was measured on specific datasets, a specific model, and a specific retrieval environment. In another domain, the optimal mix could be completely different.
It's like being told a cake needs 70% flour and 30% sugar—but what about pizza? Bread? Entirely different ratios.
The paper never answers the core question of domain transferability of reward design. Does a process reward function tuned in one domain still work in another? This directly determines how general agentic RAG can be.
Conclusion
The paper's greatest virtue is honesty. It doesn't pretend to have "solved reward design"—it clearly lays out the trade-offs of both approaches.
For engineers: don't expect a universal reward function. Be prepared to spend significant time debugging and validating reward design in your specific domain.
For researchers: Process Reward Design deserves to be a research direction in its own right—not just "how to score steps," but "how to define good research steps without importing human bias."
> "The first principle is that you must not fool yourself — and you are the easiest person to fool." When designing a reward function, the easiest person to fool is yourself.