Process vs. Outcome Reward in Agentic RAG: Lessons from a 2025 Reinforcement Learning Study

> Paper: Wenlin Zhang et al., *Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning*, arXiv:2505.14069v1, 2025-05-20. Source: https://arxiv.org/abs/2505.14069v1

Core Question

Imagine teaching a child to do research. You can either wait for the final report and grade its quality (Outcome Reward), or give feedback at every step — search, note-taking, comparison (Process Reward). Which is more effective?

This paper investigates exactly that scenario, where the "child" is an LLM and the "research" is Agentic RAG.

What the Paper Actually Does

The authors systematically compare two reinforcement learning training schemes for Agentic RAG:

• Outcome RL: Only the final answer matters. A verifier judges whether the answer is correct; a positive reward is given for a correct answer, negative for a wrong one.
• Process RL: Every step is rewarded. Generated a query? Reward. Retrieved relevant documents? Reward. Realized the retrieval was off and reformulated? Larger reward.

Feynman-style Critique: Naming ≠ Understanding

"Outcome Reward" sounds like "only the result counts," and "Process Reward" sounds like "caring about the process." Many assume that caring about the process is obviously better — if the process is right, the result follows.

But that is cargo-cult thinking.

The label "process reward" creates an illusion that the system truly cares about the process. The real danger lies elsewhere: who defines what a "good process" is?

Outcome Reward suffers from sparsity and delay, but at least "is the answer correct?" has a relatively objective verdict. Process Reward requires defining, in advance, what makes each intermediate step "good" — and that is the most dangerous part. The designer may impose their own biases on what constitutes a "good research process" and then train the model to please those biases.

Key Findings

The paper runs extensive experiments and reaches the following conclusions.

Three hard limits of Outcome RL: 1. Low exploration efficiency: The model must stumble onto many correct answers by chance, making early training extremely painful. 2. Gradient conflict: Intermediate-step gradients often point in the opposite direction of the final objective. 3. Sparse reward: A single signal may arrive only after 50 steps, creating severe credit-assignment problems.

Two dilemmas of Process RL: 1. Reward design is a dark art: How do you score "generated a query"? The goodness of a query depends on whether it ultimately helps find the right answer — but the final answer is not known at scoring time. 2. Prohibitive step-level annotation cost: Human-annotating every step is several dozen times more expensive than annotating outcomes.

Final conclusion: A hybrid reward is optimal — use process rewards to bootstrap early exploration, and outcome rewards to guarantee final quality.

The Real Insight

The most valuable takeaway is not "hybrid is best" — that is engineering common sense.

The deeper insight is that reward design in Agentic RAG is fundamentally a delayed-gratification problem. The model must balance immediate feedback (process reward) against long-term goals (outcome reward) — mirroring the human struggle to learn any complex skill.

A deeper question emerges: when an LLM's research process is sculpted by a reward function, is it actually learning to research, or learning to please the reward function? This is the reward-hacking problem. A poorly designed process reward lets the model find shortcuts — producing intermediate steps that *look* correct but contribute nothing to the final answer.

Experimental Data

Experiments were conducted on MuSiQue (multi-hop QA) and HotpotQA:

• Outcome RL performs acceptably on simple questions but lags significantly behind Process RL on multi-hop questions.
• Process RL requires carefully designed reward functions; otherwise it overfits to surface features.
• A hybrid policy (roughly 70% Process + 30% Outcome) achieves the best overall metrics.

Caveat: these experiments use simulated retrieval environments, not the real web. Under real-world retrieval, designing process rewards becomes even harder because the definition of "retrieved a good document" is itself more ambiguous.

Critical Perspective

One suspicious assumption in the paper is that an "optimal" reward ratio (70/30) exists. That ratio was measured on a specific dataset, model, and retrieval environment. In another domain, the optimal ratio could be completely different.

It is like someone telling you "a cake needs 70% flour and 30% sugar" — but what about pizza? Bread? The ratio is totally different.

The core question the paper does not answer is domain transfer of reward design. Does a Process Reward function tuned in one domain still work in another? This directly affects the generalizability of Agentic RAG.

Conclusion

The paper's greatest value is honesty. It does not pretend to have "solved reward design." Instead, it clearly lays out the trade-offs of both approaches.

For engineers, this means: do not expect a universal reward function. Plan to spend significant time tuning and validating reward design in your specific domain.

For researchers, this means: Process Reward Design deserves to be a standalone research direction — not just "how to score a step," but "how to define a good research step without introducing human bias."

> "The first principle is that you must not fool yourself — and you are the easiest person to fool." Richard Feynman. When designing reward functions, the easiest person to fool is yourself.

---

#deep-research #paper-analysis #agentic-rag #reinforcement-learning #reward-design #process-reward #outcome-reward #feynman-perspective