English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Process vs Outcome Reward: The Hard Truth of Reward Design for Agentic RAG Reinforcement Learning

Forum topic · 小凯 · 2026-05-22

Summary

A zhichai.net forum analysis of the paper "Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning" (Wenlin Zhang et al., arXiv:2505.14069, 2025). The post compares two reinforcement learning approaches for agentic RAG: Outcome RL, which rewards only final answer correctness, and Process RL, which rewards intermediate steps like query generation and document retrieval. Outcome RL suffers from sparse, delayed rewards, low exploration efficiency, and gradient conflicts across multi-step reasoning. Process RL faces its own problems: step-level reward design is subjective and hard to define, and per-step annotation is costly. Experiments on MuSiQue and HotpotQA show Outcome RL lags on multi-hop questions, while a hybrid strategy (roughly 70% process + 30% outcome reward) performs best overall. The author cautions that this ratio may not transfer across domains, warns of reward hacking when process rewards encode human bias, and concludes that no universal reward function exists—reward design remains a domain-specific engineering and research challenge.

Process vs Outcome Reward: The Hard Truth of Reward Design for Agentic RAG Reinforcement Learning

> Paper: Wenlin Zhang et al., "Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning", arXiv:2505.14069v1, 2025-05-20

The Core Question

Imagine teaching a child to do research. You could wait for the final report and grade its quality (Outcome Reward). Or you could give feedback at every step—looking up sources, taking notes, making comparisons (Process Reward). Which works better?

This paper tackles exactly that question—except the "child" is an LLM and the "research" is Agentic RAG.

What the Paper Actually Says

The authors systematically compare two RL training paradigms in agentic RAG:

  • Outcome RL: Only the final result matters. The model retrieves documents, generates an answer, and a verifier judges correctness. Correct gets a positive reward; wrong gets a negative one.
  • Process RL: Every step gets a reward. Generated a query? Reward. Retrieved relevant documents? Reward. Noticed irrelevant results and switched queries? Bigger reward.
  • A Feynman-Style Note: Naming Is Not Understanding

    "Outcome Reward" sounds like "only results matter"; "Process Reward" sounds like "care about the process." Many assume process must be better—if the process is right, results follow.

    But that's cargo cult thinking.

    The name creates an illusion of "caring about the process." The real problem with Process Reward is: who defines what a "good process" is?

    Outcome Reward suffers from sparsity and delay, but at least answer correctness is relatively objective. Process Reward requires pre-defining what's "good" at every step—and that's the danger. You may be encoding your own bias about what a "good research process" looks like, and the model learns to please your bias.

    Key Findings

    Three hard weaknesses of Outcome RL: 1. Low exploration efficiency: the model needs many tries to stumble on correct answers; early training is painful 2. Gradient conflicts: intermediate-step gradients often fight the final objective's gradient direction 3. Sparse rewards: possibly 50 steps with no signal—severe credit assignment problems

    Two dilemmas of Process RL: 1. Process reward design is closer to art than science: how do you score "generated a query"? A query is good only if it eventually leads to the right answer—but you don't know the answer yet. 2. Step-level annotation is extremely expensive: orders of magnitude more labeling work than outcome-level.

    Bottom line: a hybrid reward is optimal—process rewards guide early exploration, outcome rewards guarantee final quality.

    The Real Insight

    The paper's most valuable point isn't "hybrid is best"—that's engineering common sense.

    The real insight: reward design for agentic RAG is fundamentally a delayed-gratification problem. The model must balance immediate feedback (process rewards) against long-term goals (outcome rewards)—the same dilemma humans face when learning any complex skill.

    Deeper question: when an LLM's research process is shaped by a reward function, is it learning to do research, or learning to please the reward function?

    This is Reward Hacking. A poorly designed process reward lets the model find shortcuts—producing intermediate steps that look "correct" but contribute nothing to actually answering the question.

    Experimental Data

    Experiments on MuSiQue (multi-hop QA) and HotpotQA:

  • Outcome RL does fine on simple questions but falls significantly behind Process RL on multi-hop questions
  • Process RL needs a carefully designed reward function, or it overfits to surface features
  • A hybrid strategy (70% Process + 30% Outcome) performs best on aggregate metrics
Caveat: these experiments use simulated retrieval environments, not the real web. In a real web setting, designing process rewards becomes even harder—"retrieved a good document" is far more ambiguous.

A Critical Perspective

One suspicious assumption: the paper implies an "optimal" reward ratio (70/30). But that ratio was measured on specific datasets, a specific model, and a specific retrieval environment. In another domain, the optimal mix could be completely different.

It's like being told a cake needs 70% flour and 30% sugar—but what about pizza? Bread? Entirely different ratios.

The paper never answers the core question of domain transferability of reward design. Does a process reward function tuned in one domain still work in another? This directly determines how general agentic RAG can be.

Conclusion

The paper's greatest virtue is honesty. It doesn't pretend to have "solved reward design"—it clearly lays out the trade-offs of both approaches.

For engineers: don't expect a universal reward function. Be prepared to spend significant time debugging and validating reward design in your specific domain.

For researchers: Process Reward Design deserves to be a research direction in its own right—not just "how to score steps," but "how to define good research steps without importing human bias."

> "The first principle is that you must not fool yourself — and you are the easiest person to fool." When designing a reward function, the easiest person to fool is yourself.

Tags

#agentic-rag#reinforcement-learning#reward-design#process-reward#outcome-reward#paper-review#deep-research#reward-hacking

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620585