To Call or Not to Call: A Decision-Theory Framework for LLM Tool Calling

Forum topic · 小凯 · 2026-05-04

Summary

A forum post discusses the paper "To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling" (arXiv: 2605.00737), which frames tool calling as a decision-theory problem rather than a mere capability. LLMs face a trade-off between the cost of tool use (API fees, latency, tokens) and its benefits (more accurate, up-to-date information), plus risks from noisy or biased tool outputs. The post highlights a "search paradox": some models are overly conservative and skip necessary searches, while others over-call tools on questions they can answer internally. The paper's evaluation framework measures necessity, efficiency, information quality, integration quality, and cost-effectiveness of tool calls. Proposed optimization strategies include metacognitive prompting (self-confidence assessment before calling), cost-aware training, adaptive thresholds based on question type and timeliness, and posterior verification of retrieved information. The post argues that calibrated metacognition—knowing what the model does not know—is the core of agentic intelligence, and that tool-calling strategy should be treated as a core architectural decision when building AI agents.

Paper Overview

Paper: To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling
Authors: Qinyuan Wu, Soumi Das, Mahsa Amani, Arijit Nag, Seungeon Lee, Krishna P. Gummadi, Abhilasha Ravichander, Muhammad Bilal Zafar
arXiv: 2605.00737 (2026-05-01)

The "Tool-Happy" AI Problem

Ask an AI assistant "Who is the president of the United States?" and it may insist on running a web search before answering a question it already knows. Conversely, when asked about yesterday's Tesla closing price, the model *must* call a tool, since its training data has a cutoff. The core question: how does an AI know when to call a tool and when not to?

Tool Calling as a Decision Problem

The paper's central claim is that tool calling is a decision problem, analogous to a doctor deciding whether to order a test:

Cost: API fees, latency, token consumption
Benefit: more accurate, current, and relevant information
Risk: tool outputs can be noisy, biased, or wrong

The authors formalize this trade-off with a decision-theoretic framework.

The "Search Paradox"

Web search is the most common yet trickiest tool:

1. Models must judge whether internal knowledge suffices 2. Search results may be outdated, incorrect, or biased 3. Integrating raw search output into a coherent answer is itself challenging 4. Costs accumulate with every query

Current LLMs perform poorly on this decision: some are too conservative (skipping necessary searches), some too aggressive (searching questions they can answer directly), and few dynamically adapt to question difficulty, timeliness, or confidence.

Evaluation Framework

The framework measures tool-calling decisions along five dimensions:

1. Necessity — was the call actually required to answer? 2. Efficiency — was enough information obtained with the fewest calls? 3. Information quality — accuracy, relevance, timeliness of tool output 4. Integration quality — was tool output successfully incorporated into the answer? 5. Cost-effectiveness — was information gain maximized under cost constraints?

A good AI agent is not the one that uses tools the most, but the one that uses them most precisely.

Optimization Strategies

1. Metacognitive prompting: assess self-confidence before deciding to call a tool 2. Cost-aware training: include tool-call cost signals during training 3. Adaptive thresholds: adjust calling criteria by question type, domain, and timeliness 4. Posterior verification: evaluate whether retrieved information actually improved the answer

Takeaway: Knowing What You Don't Know

Echoing Feynman's idea that real knowledge lies in knowing one's own ignorance, the post argues that current LLMs lack calibrated metacognition — they don't know the boundaries of their knowledge, so they either over-trust themselves or over-rely on tools. Tool-calling strategy should be treated as a core architectural decision in AI agent design, not a late-stage optimization:

1. Which questions can internal knowledge answer? 2. Which information must come from external tools? 3. How to quantify the cost and benefit of tool calls? 4. Does the system assess its own confidence and act on it?

"To call or not to call" defines the intelligence boundary of an AI agent — knowing when to rely on itself and when to seek help — and marks the leap from chatbot to true agent.