The Bitter Lesson of Tool Calling: When Code Beats JSON in LLM Agents

Forum topic · 小凯 · 2026-08-29

Summary

This post analyzes the paper 'The Bitter Lesson of Tool Calling' (arXiv:2608.06370), which argues that programmatic tool calling (PTC)—where LLM agents invoke tools by writing Python code—is superior to conventional JSON-based tool calling. Framing the comparison through Rich Sutton's 'Bitter Lesson' (general methods leveraging computation outperform human knowledge injection), the author explains that JSON tool calling suffers from limited expressiveness, poor composability, and multi-turn interaction overhead, while code-based calling natively supports control flow, parallel execution, and variable binding. Experiments on BFCL v4 across 14 mainstream LLMs show PTC matches or beats JSON baselines on 11 of 14 models, with a 10.6% gain for the GPT-5.6 series and stability under context degradation where JSON baselines dropped 2.3%. Ablations indicate PTC's advantage stems from the code interface itself rather than iteration ability, and grows with newer model generations. The post also covers implementation guidance: sandboxed execution, typed function stubs with docstrings, error handling, and gradual migration strategies, while noting JSON remains preferable in safety-critical or simple scenarios.

> Paper > - Title: The Bitter Lesson of Tool Calling > - arXiv: 2608.06370 > - Authors: Ishan Patel, Sahil Sen, Elias Lumer, Vamse Kumar Subbiah > - Field: NLP / Tool Learning / Agents

Introduction: Two Craftsmen

The post opens with an allegory of two carpenters: one follows a rigid toolbox with fixed tools for fixed purposes (analogous to JSON tool calling), while the other improvises—repurposing any tool creatively for the task at hand (analogous to Programmatic Tool Calling, PTC). The debate between these paradigms mirrors the core tension identified in Rich Sutton's famous 2019 essay *The Bitter Lesson*.

Sutton's Bitter Lesson, Applied to Tool Calling

Sutton observed that over 70 years of AI history, methods relying on injected human knowledge (chess heuristics, hand-crafted visual features, linguistic syntax trees) were eventually surpassed by general methods that leverage massive computation—e.g., AlexNet beating feature engineering in 2012, Transformers displacing syntactic parsing, and AlphaZero mastering Go through pure self-play.

The paper asks whether this lesson applies to tool calling, a core LLM agent capability:

JSON tool calling: Developers predefine tools (name, parameters, description). The model emits a structured JSON string, which the system parses and executes. Strengths: predictability, safety, compatibility with existing infrastructure. Weaknesses:
Limited expressiveness — hard to express conditionals, loops, or parallel calls
Poor composability — multi-step tasks require many sequential turns (inference cost, latency, context window consumption)
Human knowledge over-injection — developers must anticipate all needed operations and workflows
Programmatic tool calling (PTC): Tools are exposed as typed Python function stubs; the model generates code that can chain, parallelize, and conditionally invoke tools within a single execution. Benefits: native control flow, natural composition, fewer interaction rounds, and extensibility to unforeseen usage patterns. Challenges: sandboxing/security, harder debugging, higher demands on model code-generation ability.

Experimental Results on BFCL v4

The paper benchmarks PTC vs. JSON across 14 mainstream LLMs on the Berkeley Function Calling Leaderboard (BFCL v4):

PTC matches or exceeds JSON on 11 of 14 models
GPT-5.6 series: +10.6% improvement over the JSON baseline
Parallel calls: PTC wins or ties on 13 of 14 models
Context Rot: under noisy, lengthened contexts, JSON baselines degrade by an average of 2.3%, while PTC remains stable

Ablation Findings

1. Iteration is not the driver: Even when JSON-based agents are given loop/iteration capability, they do not reach PTC performance. The advantage comes from the code interface itself—control flow, variable binding, composability. 2. The gap grows with model generations: Older models sometimes favor JSON; newer models benefit increasingly from PTC, suggesting PTC is a future-proof approach.

Boundaries of the Bitter Lesson

The author cautions that PTC is not a wholesale replacement for JSON:

Safety-critical scenarios (finance, medical): JSON's strict constraints may be more controllable
Simple single-call tasks: JSON's simplicity wins
Legacy integration: migration costs may be prohibitive

Practical Guidance for Adopting PTC

Sandboxed execution: container isolation (Docker, gVisor), resource limits (CPU, memory, network), syscall filtering, timeouts
Tool exposure: typed Python stubs with type annotations and docstrings (crucial for model comprehension)
Error handling: catch syntax/runtime/timeout errors, return feedback to the model for correction, cap retries
Migration strategy: start with simple tools, keep JSON as a fallback, use hybrid modes, and A/B test in production on success rate, latency, cost, and satisfaction

Conclusion: Code's Revenge

The post concludes that JSON tool calling embodies the 'human knowledge injection' pattern that Sutton warned against, while PTC embodies the general, computation-leveraging approach. Code is not replacing humans—it is liberating the AI's expressive capacity, turning the model from a rule-following tool operator into a creative problem solver.

References

Patel, I., Sen, S., Lumer, E., & Subbiah, V. K. (2026). The Bitter Lesson of Tool Calling. *arXiv preprint arXiv:2608.06370*.
Sutton, R. S. (2019). The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html