DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping
Overview
DeepPlanner (arXiv: 2510.12979, October 2025) is a research paper by Wei Fan, Wenlin Yao, Zheng Li, Feng Yao, Xin Liu, Liang Qiu, et al. (9 authors total). It targets the planning capability of deep research agents — LLM-based agents that perform long-horizon, multi-step information seeking by interleaving reasoning, search queries, and tool calls.
Key points
- Problem: In deep research tasks, the agent must decide *when* to plan, *what* to search, and *how* to coordinate retrieval with reasoning. The authors identify planning as a central bottleneck; standard RL with sparse, outcome-level rewards provides weak supervision for the planning portions of a trajectory.
- Method — Advantage Shaping: DeepPlanner trains agents with reinforcement learning and reshapes the advantage signal so that planning tokens and acting tokens receive differentiated credit. Rather than applying a uniform trajectory-level advantage, the scheme allocates advantage based on the value/novelty of the planning behavior at each step, amplifying the learning signal for effective interleaved thinking.
- Positioning: The work sits at the intersection of agentic search, retrieval-augmented generation (RAG), and RL for LLMs, where retrieval depth, tool choice, and reasoning budget are themselves learnable decisions rather than fixed pipeline stages.
- Results: The authors report improved planning quality and downstream task performance on deep research benchmarks compared with baselines. Exact datasets, metrics, and numbers should be verified against the original PDF.
- Classic search stacks separate recall, ranking, and generation; agentic approaches fold "whether and how many times to retrieve" into a learnable sequential decision process.
- Process-level credit assignment (as in DeepPlanner's advantage shaping) is an emerging pattern in agent RL, alongside process reward models and step-level supervision.
- Engineering constraints highlighted by this line of work: inference/compute budgets, multi-step latency, citation reliability, and safe behavior on the open web.
- A Comprehensive Survey of Deep Research (arXiv:2506.12594)
- A Survey of LLM-based Deep Search Agents (arXiv:2508.05668)
- Agentic Reasoning (arXiv:2502.04646)
- Original paper: DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping — https://arxiv.org/abs/2510.12979