Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
Paper: Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning Authors: Fengran Mo, Yifan Gao, Sha Li, Hansi Zeng, Xin Liu, Zhaoxuan Tan, et al. (10 authors) Published: 2026-01-19 Source: https://arxiv.org/abs/2601.13115 Category: Conversational Search
Abstract (original)
> Large Language Models (LLMs) have become a popular interface for human-AI interaction, supporting information seeking and task assistance through natural, multi-turn dialogue. To respond to users within multi-turn dialogues, the context-dependent user intent evolves across interactions, requiring contextual interpretation, query reformulation, and dynamic coordination between retrieval and generation. Existing studies usually follow static rewrite, retrieve, and generate pipelines, which optimize different procedures separately and overlook the mixed-initiative action optimization simultaneously. Although the recent developments in deep search agents demonstrate the effectiveness in jointly optimizing retrieval and generation via reasoning, these approaches focus on single-turn scenarios, which might lack the ability to handle multi-turn interactions. We introduce a conversational agent that interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training with tailored rewards towards evolving user goals. The experimental results across four widely used conversational benchmarks demonstrate the effectiveness of our methods by surpassing several strong baselines.
Key points
- Problem: User intent in multi-turn dialogue evolves across interactions, demanding contextual interpretation, query reformulation, and dynamic retrieval-generation coordination. Static rewrite → retrieve → generate pipelines optimize stages separately; deep search agents largely address only single-turn scenarios.
- Approach: A conversational agent that interleaves search and reasoning across turns, trained via reinforcement learning with rewards tailored to evolving user goals, enabling exploratory and adaptive search behaviors.
- Results: Evaluations on four widely used conversational search benchmarks show the method outperforming several existing strong baselines.
- A Survey of Conversational Search, Sep 2025, ACM
- Engineering Conversational Search Systems: A Review of Applications (arXiv:2407.00997)
- ChatRetriever: Adapting Large Language Models for Generalized and Robust (arXiv:2404.13556)
- CoSearchAgent: A Lightweight Collaborative Search Agent (arXiv:2402.06360)
- ConvGQR: Generative Query Reformulation for Conversational Search (arXiv:2305.15645)
Context within the field
The work sits at the intersection of agentic search and large-scale search/recommendation systems, addressing how to reallocate responsibilities among retrieval, ranking, generation, and tool calling in the LLM era. Notably, the number and strategy of retrieval actions themselves become learnable objects, alongside the reasoning budget.
Related directions in neural IR have progressed from BM25 through dense bi-encoders, cross-encoders, and late interaction to generative retrieval and LLM agents. RAG and agentic search extend external knowledge access from one-shot retrieval into an iterative, verifiable, plannable process—with evaluation shifting from static nDCG toward task success rate, citation accuracy, and multi-hop reasoning chain completeness.
Engineering takeaways
1. Architecture: Cascaded retrieve-rerank-generate remains mainstream, but agentic paradigms make retrieval policy itself learnable. 2. Data: High-quality instruction data and click/session logs are both critical; synthetic data requires safeguards against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Deployment: Latency, cost, interpretability, and safety are hard constraints in production—cannot optimize academic benchmarks alone.
Limitations and open questions
Typical limitations for this line of work include experimental scale constrained by compute budgets, benchmark-vs-real-user distribution mismatch, English-centric data leaving cross-lingual generalization unclear, and safety risks of open-web agent systems. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation settings.