A Practitioner's Pain Point
Imagine integrating a payment gateway: a dozen APIs, each with a dozen parameters. The standard JSON-based workflow requires round-tripping for every step—read docs, fill a JSON body, send, inspect response, retry with adjusted parameters. A dozen APIs in series become dozens of round trips, accumulating latency, errors, and frustration.
An alternative: write a single Python script that imports tool functions, loops, branches, and handles errors. N tool calls become one script execution.
This intuition was finally quantified systematically in August 2026 by Ishan Patel et al. in *The Bitter Lesson of Tool Calling* (arXiv).
Two Paradigms Compared
JSON Tool Calling
The mainstream approach. Each turn, the LLM emits a JSON payload naming a tool and parameters. The runtime executes the call, returns the result, and the LLM decides the next step. N tool calls = N round trips.
Programmatic Tool Calling (PTC)
The paper's advocated approach. Instead of emitting JSON, the LLM writes a Python script that imports tool functions, loops, parallelizes, and handles exceptions. N tool calls = 1 script execution.
The key distinction: in JSON mode the LLM is a dispatcher that returns for every call; in PTC mode the LLM is a programmer that writes the code and exits, leaving execution to the interpreter.
BFCL v4 Benchmark: 14-Model Cross-Evaluation
The authors evaluated PTC vs JSON tool calling on BFCL (Berkeley Function Calling Leaderboard) v4 across 14 models, including GPT-4o, GPT-5, GPT-5.6 (Luna/Sol/Terra), GPT-5-nano, Claude Sonnet 4.5, and Claude Sonnet 5.
Headline Results
- PTC matched or beat JSON on 11 of 14 models.
- GPT-5.6 family averaged a 10.6% improvement with PTC.
- GPT-4o era: PTC lagged JSON by 26.9%—the model could not yet write code reliably.
- GPT-5-nano: PTC reached parity—an inflection point.
- GPT-5.6 era: PTC converged with JSON and began to surpass it.
- PTC cut chained-call wall-clock time roughly in half on 13 of 14 models, with per-call latency ratios ranging from 0.5 to 0.9.
- This reflects a paradigm difference: N round trips vs 1 execution.
- JSON tool calling hits a hard structural ceiling at N=70–72 for Claude Sonnet 5: the model simply stops emitting calls—not errors, just omissions—as context grows.
- PTC maintained 100% enumeration accuracy at N=100, because a for-loop scales with iteration count, not LLM context.
- PTC matched or beat JSON on fan-out for 13 of 14 models.
- JSON tool calling degraded by an average of 2.3% under context rot.
- PTC remained stable.
- JSON tool calling is the engineered approach: humans design schemas, parameter types, and protocols for the model to operate within.
- PTC is the general-compute approach: the model writes code; code is computation; computation is general.
- N calls = N LLM inferences = N latencies
- State for all N calls sits in LLM context → context bloat
- Context bloat → context rot → dropped calls
- N calls = 1 script execution = 1 latency
- State lives in script-local variables → no context bloat
- No bloat → no rot → no dropped calls
- PTC requires strong code ability. GPT-4o-era models performed worse with PTC; JSON remains preferable for small or code-weak models.
- Security is an open question. Executing LLM-written code grants whatever privileges code can exercise. JSON's schema provides natural constraints that PTC lacks. Sandboxing, permissions, and auditing all need redesign.
- BFCL v4 is a specific benchmark. Whether PTC's advantages generalize to multi-turn dialogue, tool composition, or error recovery remains to be validated.
- Euclid-MCP: LLM as poet, Prolog as accountant—offloading reasoning to specialized tools.
- colibrì: 1,300 lines of C running a 744B-parameter MoE model on a 25GB laptop without a GPU, decoupling total parameters from memory by ~15×.
- Rebucca: small model filters, large model verifies—division of labor over unification.
- Paper: https://arxiv.org/abs/2608.06370
- HTML: https://arxiv.org/html/2608.06370v1
- Related project: https://github.com/cameronking4/programmatic-tool-calling-ai-sdk
Evolution Across Model Generations
The authors label this phenomenon *The Bitter Lesson of Tool Calling*, echoing Rich Sutton's classic essay: general methods leveraging computation ultimately outperform engineered approaches built on human knowledge.
Three Structural Ablations
1. Chained Calls
Chaining is the pattern A → B → C, where each call's result feeds the next.
2. Parallel Fan-Out
Parallel fan-out means invoking N independent tools simultaneously.
Insight: JSON tool calling has a structural ceiling tied to context length; PTC does not. The ceiling's exact position varies by model (GPT-4o's was lower than Claude Sonnet 5's), but the ceiling itself is inherent to any "return-to-LLM-per-call" design.
3. Context Rot
In long conversations, early tool-call results degrade—models misread, ignore, or confuse older context.
The reason: in PTC, prior call results live in script-local variables, not in the LLM's context. Only the script itself occupies the LLM context. PTC is structurally immune to this form of context rot.
Why Call It the Bitter Lesson
Sutton's *The Bitter Lesson* argues that general methods based on search and learning ultimately beat engineered approaches based on human knowledge—a thesis validated in Go (AlphaGo), translation (NMT), and dialogue (GPT).
Applied to tool calling:
The evolutionary data is striking: PTC lost to JSON in the GPT-4o era because models could not write code; PTC surpassed JSON in the GPT-5.6 era because they can. This matches Sutton's prediction: once models become capable enough, general methods overtake engineered ones. JSON schemas were a compromise for incapable models; once models can code, that compromise becomes a structural burden.
A Deeper Structural Insight
The most memorable contribution is not a specific number but a paradigm-level difference:
JSON mode (LLM as dispatcher):
PTC mode (LLM as programmer):
This is not an optimization; it is a paradigm shift. The dispatcher paradigm has a structural ceiling; the programmer paradigm does not.
The author draws a cross-domain analogy to octopus neural systems—DNA encodes construction rules rather than every synapse, leaving RNA editing to assemble wiring at runtime. PTC mirrors this: LLM pretraining writes the script; the interpreter handles execution. Division of labor beats unification. Letting the LLM do what it is good at (writing code) and the interpreter do what it is good at (executing code) is more efficient than forcing the LLM to do everything.
Honest Limitations
These caveats do not undermine the core contribution: systematically quantifying PTC vs JSON, exposing JSON's structural ceiling, and predicting how the bitter lesson will play out in tool calling.
Resonance With Other Recent Work
The article links the paper to several other recent works:
All four point to a recurring principle: specialized tools for specialized tasks, mirroring biological division (octopus DNA/RNA), engineering division (CPU/GPU), and AI division (LLM/interpreter).
Closing
The sentence worth keeping: once models become capable enough, general methods overtake engineered ones.
JSON tool calling is a compromise made when models could not write code. Once they can, the compromise becomes a structural ceiling. PTC is not an optimization—it is a paradigm shift from "LLM as dispatcher" to "LLM as programmer."
The consequences are only beginning to surface. When the LLM no longer needs to participate in every call, the cost, latency, and reliability of tool use will all be redefined. Engineering efforts that pile more features onto JSON schemas may be repeating the bitter lesson Sutton described.
---