Inference-Time Budget Control for LLM Search Agents

Overview

This forum post introduces a research paper on inference-time budget control for LLM search agents, indexed under the Agentic Search section of the forum's paper collection.

| Field | Detail | |-------|--------| | Title | Inference-Time Budget Control for LLM Search Agents | | Authors | Zhengru Fang, Senkang Forest Hu, Zhonghao Chang, Yu Guo, Yihang Tao, Hongyao Liu, et al. (9 authors) | | Link | https://arxiv.org/abs/2605.05701 | | Type | Academic paper |

Problem Statement

LLM search agents increasingly rely on tools at inference time, but their trajectories are often constrained by hard limits on both tool calls and generated tokens. Under such dual budgets, better answers require not only stronger models, but also explicit control over which search action should receive the next budget unit, and when the accumulated evidence is sufficient to commit a final answer. The paper studies this in multi-hop question answering (QA) and formulates it as a two-stage inference-time budget control problem.

Method

1. Search-time control: A controller assigns each feasible action a task-level Value-of-Information (VOI) score — an operational estimate of marginal task value per unit budget under the current search state and remaining dual budget. This score governs the choice among retrieval, decomposition, and answer commitment. 2. Answer-time control: A selective, evidence-grounded finalizer compares the trajectory answer with a refined candidate and rewrites only when the residual error appears to be a low-risk answer-form error.

Results

• Evaluated across four multi-hop QA benchmarks, three LLM backbones, and four budget levels.
• The method yields positive aggregate gains over four audited baselines under the same hard dual-budget protocol.
• Ablations: search-time budget control — especially the budget-dependent penalty — provides the main performance gain; answer-time control helps mainly when the retrieval path is already adequate.
• Conclusion: inference-time budget control should govern both how budget is spent during search and how the final answer is committed.

Key Takeaways for Search/Rec/Personalization

• Architecture: Cascaded retrieve–rerank–generate remains mainstream, but the agentic paradigm makes retrieval count and policy itself a learnable/controllable object.
• Evaluation: Offline metrics diverge from online satisfaction; process-level metrics (task success, citation accuracy, reasoning-chain completeness) are increasingly important.
• Deployment: Latency, cost, interpretability, and safety are hard constraints for industrial adoption — do not optimize academic benchmarks alone.
• Open problems: benchmark trustworthiness, latency/cost trade-offs, hallucination and safety, cross-lingual and multimodal extension.

Engineering Checklist

| Item | Question | Recommendation | |------|----------|----------------| | Data | PII in training/index? Versioning? | Partitioned indexes, sanitization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascades + early stopping, query caching, async reranking | | Quality | Does offline gain transfer online? | Interleaving experiments, human audits, citation checks | | Safety | Poisoning/bias from open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Token & GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense retrieval |

Glossary

• RAG — Retrieval-Augmented Generation
• Agentic Search — modeling search as sequential decision-making and tool invocation
• VOI — Value-of-Information, marginal task value per unit budget
• nDCG — ranking-quality metric

Original Abstract

> LLM search agents increasingly rely on tools at inference time, but their trajectories are often constrained by hard limits on both tool calls and generated tokens. Under such dual budgets, better answers require not only stronger models, but also explicit control over which search action should receive the next budget unit and when the accumulated evidence is sufficient to commit a final answer. We study this problem in multi-hop question answering (QA) and formulate it as two-stage inference-time budget control. At search time, our controller assigns each feasible action a task-level Value-of-Information (VOI) score, defined as an operational estimate of marginal task value per unit budget under the current search state and remaining dual budget, and uses this score to choose among retrieval, decomposition, and answer commitment. After search, a selective evidence-grounded finalizer compares the trajectory answer with a refined candidate and rewrites only when the residual error appears to be a low-risk answer-form error. Across four multi-hop QA benchmarks, three LLM backbones, and four budget levels, the method yields positive aggregate gains over four audited baselines under the same hard dual-budget protocol. Ablations show that search-time budget control, especially budget-dependent penalty, provides the main performance gain, while answer-time control helps mainly when the retrieval path is already adequate. These results suggest that inference-time budget control for LLM search agents should govern both how budget is spent during search and how the final answer is committed.

Related Entries

• A Systematic Framework for Enterprise Knowledge Retrieval
• Synergizing RAG and Reasoning: A Systematic Review
• AceSearcher: Bootstrapping Reasoning and Search for LLMs
• Agentic Information Retrieval