DecoupleSearch: Decouple Planning and Search via Hierarchical Reward Modeling
Overview
| Field | Content | |------|------| | Title | DecoupleSearch: Decouple Planning and Search via Hierarchical Reward Modeling | | Authors | Hao Sun, Zile Qiao, Bo Wang, Guoxin Chen, Yingyan Hou, Yong Jiang, et al. (9 authors) | | Published | 2025-09-07 | | Link | https://arxiv.org/abs/2510.21712 | | Type | Academic paper | | Category | Agentic Search |
Background
Agentic RAG introduces autonomous agents into the RAG workflow to improve flexibility, but faces several challenges:
1. The success of each step depends on both high-quality planning and accurate search; 2. There is a lack of supervision for intermediate reasoning steps; 3. The candidate space for planning and searching grows exponentially large.
DecoupleSearch is proposed to address these challenges by treating planning and search as separately optimizable processes.
Core Contribution
The framework decouples planning and search processes using dual value models, enabling independent optimization of plan reasoning and search grounding.
Method
- Reasoning tree: constructs a tree where each node represents a planning or search step.
- Monte Carlo Tree Search (MCTS): used to assess the quality of each step, providing supervision signal for intermediate reasoning.
- Hierarchical Beam Search: during inference, planning and search candidates are iteratively refined using the dual value models.
- The agentic paradigm is turning "how many times to retrieve and with what strategy" into a learnable object, beyond the classic cascade of retrieval → rerank → generation.
- Decoupling planning from search allows each component to be optimized independently, mitigating the lack of intermediate supervision.
- MCTS-based step quality assessment provides a principled way to supervise multi-step reasoning in search agents.
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforcement Learning
- Agentic Information Retrieval (arXiv:2410.09713)
- Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
- Original paper: DecoupleSearch: Decouple Planning and Search via Hierarchical Reward Modeling. https://arxiv.org/abs/2510.21712
The technique follows the standard pipeline of problem formalization → model/system design → training procedure → inference pipeline, with the dual value models scoring plan quality and search grounding separately.
Original Abstract
> Retrieval-Augmented Generation (RAG) systems have emerged as a pivotal methodology for enhancing Large Language Models (LLMs) through the dynamic integration of external knowledge. To further improve RAG's flexibility, Agentic RAG introduces autonomous agents into the workflow. However, Agentic RAG faces several challenges: (1) the success of each step depends on both high-quality planning and accurate search, (2) the lack of supervision for intermediate reasoning steps, and (3) the exponentially large candidate space for planning and searching. To address these challenges, we propose DecoupleSearch, a novel framework that decouples planning and search processes using dual value models, enabling independent optimization of plan reasoning and search grounding. Our approach constructs a reasoning tree, where each node represents planning and search steps. We leverage Monte Carlo Tree Search to assess the quality of each step. During inference, Hierarchical Beam Search iteratively refines planning and search candidates with dual value models. Extensive experiments across policy models of varying parameter sizes demonstrate the effectiveness of our method.
Evaluation
Experiments were conducted across policy models of varying parameter sizes. Specific numerical results should be verified against the original PDF tables.