An Engineer's Frustration
Suppose you are a backend engineer asked to integrate a payment gateway. The SDK docs list a dozen APIs, each with a dozen parameters. Your workflow:
1. Read the API docs 2. Fill in a JSON request body 3. Send the request 4. Inspect the response 5. If it fails, fill in another JSON based on the error code 6. Send again 7. Repeat until success
Every step is a round trip. Chain a dozen APIs and you get dozens of round trips. Latency accumulates, errors accumulate, and so does your impatience.
Now switch paradigms: write a Python script that chains all the API calls. One execution, all calls complete in the same process — parallelism, error handling, and branching all controlled by code. Latency drops from "dozens of round trips" to "one execution."
Which is faster? Obvious. Yet nobody systematically quantified it until August 2026, when Ishan Patel et al. published *The Bitter Lesson of Tool Calling* on arXiv.
Two Tool-Calling Paradigms
JSON Tool Calling
The current mainstream. Each turn, the LLM outputs a JSON structure describing which tool to call and with what arguments. The system executes it, appends the result to context, and the LLM decides the next step.
Every step is a full LLM inference + external call + context update. N tool calls = N round trips.
Programmatic Tool Calling (PTC)
The paper's advocated approach. The LLM doesn't emit JSON — it writes a Python script that imports tool functions, loops, calls in parallel, and handles exceptions. N tool calls = 1 script execution.
The key difference: in JSON mode the LLM is a dispatcher — every call returns to the LLM; in PTC mode the LLM is a programmer — it writes the code and exits, leaving execution to the interpreter.
This looks like mere engineering optimization, but it exposes a deeper structural problem.
A 14-Model Bake-off on BFCL v4
The paper benchmarks 14 models on BFCL (Berkeley Function Calling Leaderboard) v4, spanning GPT-4o through the GPT-5.6 Luna/Sol/Terra family, and Claude Sonnet 4.5 through Claude Sonnet 5.
Main results
PTC matches or exceeds JSON tool calling on 11 of 14 models. The GPT-5.6 family improves by 10.6% on average.
The more notable finding is the evolutionary trend:
- GPT-4o era: PTC trails JSON by 26.9% — models couldn't yet write good code
- GPT-5-nano: PTC first catches up — a capability inflection point
- GPT-5.6 era: PTC fully converges and starts to overtake
- JSON tool calling = the engineered approach: humans designed schemas, parameter types, calling protocols
- PTC = the compute-based general approach: let the model write code
- N calls = N inferences = N latencies
- All call state lives in context = context bloat
- Context bloat → context rot → dropped calls
- N calls = 1 script execution = 1 latency
- Call state lives in local variables = no context bloat
- No bloat → no rot → no dropped calls
- Euclid-MCP: let the LLM be the poet, Prolog the accountant — reasoning outsourced to a specialized tool, mirroring PTC outsourcing scheduling to an interpreter.
- colibrì: 744B-parameter MoE inference in 1,300 lines of C on a 25GB no-GPU laptop, decoupling parameter count from memory; PTC decouples call count from inference count.
- Rebucca: small model triage + large model verification — division of labor again.
The authors name this "The Bitter Lesson of Tool Calling," echoing Rich Sutton's classic. Sutton's thesis: compute-based general methods eventually beat methods built on human-engineered knowledge. Here: letting models write code (general computation) eventually beats making them fill JSON (engineered structure).
Three Structural Ablations
1. Chaining
Chaining: call A → take result → call B → take result → call C. JSON mode returns to the LLM each step; PTC does it in one script.
PTC cuts chained-call wall-clock time in half on 13/14 models, with per-item latency ratios from 0.5 to 0.9. This isn't an optimization — it's a paradigm difference: N round trips vs 1 execution.
2. Parallel Fan-Out
Fan-out: call N independent tools simultaneously. The paper finds a hard structural limit:
JSON tool calling completely drops tool calls at N=70–72 (Claude Sonnet 5) — not errors, just no calls at all. The context grows so long the model "forgets" which tools to call.
PTC maintains 100% enumeration accuracy at N=100, because it's just a for loop.
PTC matches or beats JSON fan-out on 13/14 models.
Key insight: JSON mode has a hard structural ceiling; PTC doesn't. The ceiling moves with model capability, but it is structural — as long as every call returns to the LLM, context length imposes a limit.
3. Context Rot
Context rot: over long conversations, earlier tool-call results "rot" — the model misreads, ignores, or confuses old context.
JSON tool calling degrades by an average of 2.3% under context rot. PTC stays stable.
Why? In PTC, old call results don't live in LLM context — they live in script local variables. The LLM context holds only the script, not each call's result. PTC is structurally immune to context rot.
Why "Bitter Lesson"
Sutton's core claim: general methods based on search and learning ultimately beat human-knowledge-based engineering — validated in Go (AlphaGo), translation (NMT), dialogue (GPT).
This paper applies it to tool calling:
The evolution data is compelling: in the GPT-4o era PTC lost because models couldn't write code; in the GPT-5.6 era PTC wins because they can. JSON mode was a compromise for weak models; once models code well, the compromise becomes a liability.
A Deeper Structural Insight
The most memorable takeaway isn't any number, but the structural difference:
In JSON mode, the LLM is a dispatcher — every call returns to the LLM. Consequences:
In PTC mode, the LLM is a programmer — it writes the code and exits. Consequences:
This is paradigm-level, not optimization-level. The dispatcher paradigm has a structural ceiling; the programmer paradigm doesn't.
The post draws a cross-domain analogy: an octopus's "DNA pre-training + RNA inference-time computation" — the genome encodes construction rules, and RNA editing adjusts connections in real time. PTC is isomorphic: LLM pre-training (writes the script) + interpreter inference-time computation (runs it).
The general engineering principle: division of labor beats monolithic control. Let the LLM do what it's good at (writing code), let the interpreter do what it's good at (executing code).
Honest Assessment
The paper has limitations:
1. PTC demands strong coding ability. GPT-4o underperformed, so JSON may remain better for small or code-weak models. 2. Safety is an open question. Executing LLM-written scripts lets the model do anything code can do. JSON schemas provide natural constraints; PTC doesn't. Sandboxing, permissions, and auditing need redesign. 3. BFCL v4 is one benchmark. Whether PTC's advantage generalizes to multi-turn dialogue, tool composition, and error recovery needs more validation.
But the core contribution stands: a systematic quantification of PTC vs JSON, a demonstration of JSON's structural ceiling, and a roadmap for the Bitter Lesson arriving in tool calling.
Cross-Paper Resonance
The post links this paper to several recent works:
Conclusion
The most memorable line: when model capability is strong enough, general methods overtake engineered methods. JSON tool calling was a compromise for weak models; it has become a structural ceiling. PTC is not an optimization — it is a paradigm switch from "LLM as dispatcher" to "LLM as programmer," with consequences for cost, latency, and reliability only beginning to unfold.
---
Paper: https://arxiv.org/abs/2608.06370 HTML full text: https://arxiv.org/html/2608.06370v1 Related project: https://github.com/cameronking4/programmatic-tool-calling-ai-sdk