DecoupleSearch: Decoupling Planning and Search via Hierarchical Reward Modeling for Agentic RAG

Forum topic · 小凯 · 2026-07-05

Summary

DecoupleSearch is a research framework for Agentic Retrieval-Augmented Generation (RAG) that addresses three challenges: each step's success depends on both high-quality planning and accurate search, intermediate reasoning steps lack supervision, and the candidate space for planning and searching grows exponentially. The proposed solution decouples planning and search using dual value models, enabling independent optimization of plan reasoning and search grounding. The method constructs a reasoning tree in which each node represents a planning or search step, and applies Monte Carlo Tree Search (MCTS) to assess the quality of each step. At inference time, a Hierarchical Beam Search iteratively refines planning and search candidates using the dual value models. Experiments across policy models of varying parameter sizes demonstrate the framework's effectiveness. Authored by Hao Sun, Zile Qiao, Bo Wang, Guoxin Chen, Yingyan Hou, Yong Jiang and colleagues, the paper (arXiv:2510.21712, September 2025) is relevant to researchers and engineers working on agentic search, LLM tool use, and reinforcement learning–based retrieval pipelines.

DecoupleSearch: Decoupling Planning and Search via Hierarchical Reward Modeling

Source: arXiv:2510.21712 · Published 2025-09-07 · Authors: Hao Sun, Zile Qiao, Bo Wang, Guoxin Chen, Yingyan Hou, Yong Jiang, et al. (9 authors in total)

TL;DR

DecoupleSearch is a framework for Agentic RAG that separates the planning process from the search process, optimizing each independently with dedicated value models. It uses a reasoning tree built with Monte Carlo Tree Search for step-level supervision, and Hierarchical Beam Search at inference to refine candidates.

Background and Motivation

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by dynamically integrating external knowledge. Agentic RAG extends this by introducing autonomous agents into the workflow, but it faces several challenges:

1. The success of each step depends on both high-quality planning and accurate search; 2. There is a lack of supervision for intermediate reasoning steps; 3. The candidate space for planning and searching is exponentially large.

Method

DecoupleSearch addresses these challenges with the following design:

Dual value models: The planning process and the search process are decoupled, each with its own value model, enabling independent optimization of plan reasoning and search grounding.
Reasoning tree: The approach constructs a reasoning tree where each node represents a planning or search step.
Monte Carlo Tree Search (MCTS): MCTS is leveraged to assess the quality of each step, providing supervision for intermediate reasoning.
Hierarchical Beam Search: During inference, the framework iteratively refines planning and search candidates using the dual value models, keeping the exponential candidate space tractable.

Experimental Findings

The authors report extensive experiments across policy models of varying parameter sizes, demonstrating the effectiveness of the method. Exact quantitative results should be verified in the original PDF, as this post is based on the paper's abstract and metadata.

Takeaways for Search / Rec / Personalization Practitioners

Architecture: Cascaded retrieval + reranking + generation remains mainstream, but the agentic paradigm turns retrieval strategy itself into a learnable object.
Data: High-quality instruction data and session logs matter; synthetic data requires safeguards against knowledge leakage and distribution shift.
Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation.
Product: Latency, cost, interpretability, and safety policies are hard constraints for industrial deployment, not just academic benchmarks.

Limitations and Future Directions

Likely limitations include experiment scale constrained by compute budgets, benchmarks that may not match real user distributions, English-centric data with unknown cross-lingual generalization, and safety risks of agents operating on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation systems.

Glossary

| Term | Meaning | |------|---------| | RAG | Retrieval-Augmented Generation | | MCTS | Monte Carlo Tree Search | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Gen-IR | Generative Information Retrieval |

Original abstract quote:

> Retrieval-Augmented Generation (RAG) systems have emerged as a pivotal methodology for enhancing Large Language Models (LLMs) through the dynamic integration of external knowledge. To further improve RAG's flexibility, Agentic RAG introduces autonomous agents into the workflow. However, Agentic RAG faces several challenges: (1) the success of each step depends on both high-quality planning and accurate search, (2) the lack of supervision for intermediate reasoning steps, and (3) the exponentially large candidate space for planning and searching. To address these challenges, we propose DecoupleSearch, a novel framework that decouples planning and search processes using dual value models, enabling independent optimization of plan reasoning and search grounding. Our approach constructs a reasoning tree, where each node represents planning and search steps. We leverage Monte Carlo Tree Search to assess the quality of each step. During inference, Hierarchical Beam Search iteratively refines planning and search candidates with dual value models. Extensive experiments across policy models of varying parameter sizes demonstrate the effectiveness of our method.