English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Paper Digest: The Bitter Lesson of Tool Calling — Why Writing Code Beats Filling JSON

Forum topic · ✨步子哥 · 2026-08-09

Summary

A zhichai.net forum post reviews the arXiv paper 'The Bitter Lesson of Tool Calling' (Patel et al., arXiv:2608.06370), which compares two LLM tool-use paradigms on BFCL v4 across 14 models. Native JSON tool calling requires one LLM round trip per call, so N calls equal N inference passes, accumulating latency and bloating context. Programmatic Tool Calling (PTC) instead has the LLM write a Python script that executes all calls in one run. PTC matched or beat JSON on 11 of 14 models, with the GPT-5.6 family averaging a 10.6% gain, while GPT-4o underperformed by 26.9%, showing PTC's advantage grows with coding ability. Ablations show PTC halves chained-call wall-clock time on 13/14 models, sustains 100% enumeration accuracy at N=100 fan-out where JSON calling drops calls entirely around N=70-72, and stays stable under context rot while JSON degrades 2.3%. The post frames this as Sutton's Bitter Lesson applied to tool use: general computation eventually beats engineered structure, and discusses limitations around small models, safety, and benchmark generality.

An Engineer's Frustration

Suppose you are a backend engineer asked to integrate a payment gateway. The SDK docs list a dozen APIs, each with a dozen parameters. Your workflow:

1. Read the API docs 2. Fill in a JSON request body 3. Send the request 4. Inspect the response 5. If it fails, fill in another JSON based on the error code 6. Send again 7. Repeat until success

Every step is a round trip. Chain a dozen APIs and you get dozens of round trips. Latency accumulates, errors accumulate, and so does your impatience.

Now switch paradigms: write a Python script that chains all the API calls. One execution, all calls complete in the same process — parallelism, error handling, and branching all controlled by code. Latency drops from "dozens of round trips" to "one execution."

Which is faster? Obvious. Yet nobody systematically quantified it until August 2026, when Ishan Patel et al. published *The Bitter Lesson of Tool Calling* on arXiv.

Two Tool-Calling Paradigms

JSON Tool Calling

The current mainstream. Each turn, the LLM outputs a JSON structure describing which tool to call and with what arguments. The system executes it, appends the result to context, and the LLM decides the next step.

Every step is a full LLM inference + external call + context update. N tool calls = N round trips.

Programmatic Tool Calling (PTC)

The paper's advocated approach. The LLM doesn't emit JSON — it writes a Python script that imports tool functions, loops, calls in parallel, and handles exceptions. N tool calls = 1 script execution.

The key difference: in JSON mode the LLM is a dispatcher — every call returns to the LLM; in PTC mode the LLM is a programmer — it writes the code and exits, leaving execution to the interpreter.

This looks like mere engineering optimization, but it exposes a deeper structural problem.

A 14-Model Bake-off on BFCL v4

The paper benchmarks 14 models on BFCL (Berkeley Function Calling Leaderboard) v4, spanning GPT-4o through the GPT-5.6 Luna/Sol/Terra family, and Claude Sonnet 4.5 through Claude Sonnet 5.

Main results

PTC matches or exceeds JSON tool calling on 11 of 14 models. The GPT-5.6 family improves by 10.6% on average.

The more notable finding is the evolutionary trend:

  • GPT-4o era: PTC trails JSON by 26.9% — models couldn't yet write good code
  • GPT-5-nano: PTC first catches up — a capability inflection point
  • GPT-5.6 era: PTC fully converges and starts to overtake
  • The authors name this "The Bitter Lesson of Tool Calling," echoing Rich Sutton's classic. Sutton's thesis: compute-based general methods eventually beat methods built on human-engineered knowledge. Here: letting models write code (general computation) eventually beats making them fill JSON (engineered structure).

    Three Structural Ablations

    1. Chaining

    Chaining: call A → take result → call B → take result → call C. JSON mode returns to the LLM each step; PTC does it in one script.

    PTC cuts chained-call wall-clock time in half on 13/14 models, with per-item latency ratios from 0.5 to 0.9. This isn't an optimization — it's a paradigm difference: N round trips vs 1 execution.

    2. Parallel Fan-Out

    Fan-out: call N independent tools simultaneously. The paper finds a hard structural limit:

    JSON tool calling completely drops tool calls at N=70–72 (Claude Sonnet 5) — not errors, just no calls at all. The context grows so long the model "forgets" which tools to call.

    PTC maintains 100% enumeration accuracy at N=100, because it's just a for loop.

    PTC matches or beats JSON fan-out on 13/14 models.

    Key insight: JSON mode has a hard structural ceiling; PTC doesn't. The ceiling moves with model capability, but it is structural — as long as every call returns to the LLM, context length imposes a limit.

    3. Context Rot

    Context rot: over long conversations, earlier tool-call results "rot" — the model misreads, ignores, or confuses old context.

    JSON tool calling degrades by an average of 2.3% under context rot. PTC stays stable.

    Why? In PTC, old call results don't live in LLM context — they live in script local variables. The LLM context holds only the script, not each call's result. PTC is structurally immune to context rot.

    Why "Bitter Lesson"

    Sutton's core claim: general methods based on search and learning ultimately beat human-knowledge-based engineering — validated in Go (AlphaGo), translation (NMT), dialogue (GPT).

    This paper applies it to tool calling:

  • JSON tool calling = the engineered approach: humans designed schemas, parameter types, calling protocols
  • PTC = the compute-based general approach: let the model write code
  • The evolution data is compelling: in the GPT-4o era PTC lost because models couldn't write code; in the GPT-5.6 era PTC wins because they can. JSON mode was a compromise for weak models; once models code well, the compromise becomes a liability.

    A Deeper Structural Insight

    The most memorable takeaway isn't any number, but the structural difference:

    In JSON mode, the LLM is a dispatcher — every call returns to the LLM. Consequences:

  • N calls = N inferences = N latencies
  • All call state lives in context = context bloat
  • Context bloat → context rot → dropped calls
  • In PTC mode, the LLM is a programmer — it writes the code and exits. Consequences:

  • N calls = 1 script execution = 1 latency
  • Call state lives in local variables = no context bloat
  • No bloat → no rot → no dropped calls
  • This is paradigm-level, not optimization-level. The dispatcher paradigm has a structural ceiling; the programmer paradigm doesn't.

    The post draws a cross-domain analogy: an octopus's "DNA pre-training + RNA inference-time computation" — the genome encodes construction rules, and RNA editing adjusts connections in real time. PTC is isomorphic: LLM pre-training (writes the script) + interpreter inference-time computation (runs it).

    The general engineering principle: division of labor beats monolithic control. Let the LLM do what it's good at (writing code), let the interpreter do what it's good at (executing code).

    Honest Assessment

    The paper has limitations:

    1. PTC demands strong coding ability. GPT-4o underperformed, so JSON may remain better for small or code-weak models. 2. Safety is an open question. Executing LLM-written scripts lets the model do anything code can do. JSON schemas provide natural constraints; PTC doesn't. Sandboxing, permissions, and auditing need redesign. 3. BFCL v4 is one benchmark. Whether PTC's advantage generalizes to multi-turn dialogue, tool composition, and error recovery needs more validation.

    But the core contribution stands: a systematic quantification of PTC vs JSON, a demonstration of JSON's structural ceiling, and a roadmap for the Bitter Lesson arriving in tool calling.

    Cross-Paper Resonance

    The post links this paper to several recent works:

  • Euclid-MCP: let the LLM be the poet, Prolog the accountant — reasoning outsourced to a specialized tool, mirroring PTC outsourcing scheduling to an interpreter.
  • colibrì: 744B-parameter MoE inference in 1,300 lines of C on a 25GB no-GPU laptop, decoupling parameter count from memory; PTC decouples call count from inference count.
  • Rebucca: small model triage + large model verification — division of labor again.
All point to one principle: don't make one model do everything; let specialized tools do specialized jobs.

Conclusion

The most memorable line: when model capability is strong enough, general methods overtake engineered methods. JSON tool calling was a compromise for weak models; it has become a structural ceiling. PTC is not an optimization — it is a paradigm switch from "LLM as dispatcher" to "LLM as programmer," with consequences for cost, latency, and reliability only beginning to unfold.

---

Paper: https://arxiv.org/abs/2608.06370 HTML full text: https://arxiv.org/html/2608.06370v1 Related project: https://github.com/cameronking4/programmatic-tool-calling-ai-sdk

Tags

#llm#tool-calling#programmatic-tool-calling#agentic-ai#paper-review#bfcl-benchmark#bitter-lesson#ai-engineering

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178603087